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EVAL  USERS  MANUAL 


1.0  INTRODUCTION 


1.1  Purpose  of  the  Users  Manual 


The  purpose  of  this  manual  is  to  provide  users  of  the 
EVAL  system  with  the  background  material  and  the  detailed 
instructions  necessary  to  use  and  interpret  the  various 
functions  that  EVAL  provides.  The  manual  also  presents  the 
decision-analytic  concepts  inherent  in  the  EVAL  approach, 
including  the  assumptions  and  restrictions  concerning  its 
use.  The  manual  includes  case  study  applications. 


Because  the  manual  must  serve  users  both  skilled  and 
unskilled  in  the  use  of  decision-analytic  methodology,  it  is 
prepared  in  a  modular  fashion.  Thus,  whereas  the  initial 
sections  provide  detailed  information  for  the  naive  user, 
the  last  section  is  direct  and  unelaborated  for  those  users 
knowledgeable  in  the  approach,  hc _  _ _ _ _ 


1.2  References 

1.2.1  Barclay,  Scott,  et  al.  Handbook  for  Decision 
Analysis.  Technical  Report  77-6-30.  McLean, 
Virginia:  Decisions  and  Designs,  Inc.,  Sep¬ 
tember  1977. 

1.2.2  Gulick,  Roy  M.  Documentation  of  Decision- 
Aiding  Software:  Introductory  Guide.  Tech¬ 
nical  Report  TR  79-1-93.  McLean,  Virginia: 
Decisions  and  Designs,  Inc.,  in  press. 
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1.2.3  Allardyce,  Linda  B.;  Amey,  Dorothy  M. ?  Feuerwerger 
Phillip  H.;  Gulick,  Roy  M.  Documentation  of 
Decision-Aiding  Software:  EVAL  Functional 


Description.  McLean,  Virginia:  Decisions  and 
Designs,  Inc.,  November  1979. 

1.2.4  Allardyce,  Linda  B.;  Amey,  Dorothy  M. ;  Feuerwerger 
Phillip  H.;  Gulick,  Roy  M.  Documentation  of 
Decision-Aiding  Software:  EVAL  System  Specifica¬ 
tion.  McLean,  Virginia:  Decisions  and  Designs 
Inc.,  November  1979. 

1.3  Abbreviations  and  Terms 

1.3.1  EVAL  -  EVAL,  the  name  of  the  system,  is  an 
abbreviation  for  Evaluation,  reflecting  the  system's  major 
area  of  applicability. 


1.3.2  Terms  -  Standard  mathematical  notations  and 
decision-analytic  terminology  are  used  throughout  this 
manual.  Decision-analytic  terms  are  defined  when  they  are 
first  encountered.  Reference  1.2.1  provides  more  detail  on 
decision  analysis,  should  it  be  desired. 
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2.0  SYSTEM  SUMMARY 


2 . 1  Background 


Military  decision  makers  and  their  staffs  often  must 
evaluate  several  alternative  future  courses  of  action  and 
choose  the  one  course  that  appears  to  be  optimal.  In  most 
important  decision  situations  that  choice  is  made  difficult 
by  two  factors:  the  uncertainty  of  events  and  the  multiple 
attributes  of  the  postulated  decision  outcomes. 

Uncertainty  manifests  itself  by  the  presence  of  key 
future  events  that  will  eventually  impact  on  the  decision 
but  whose  manner  of  unfolding  is  beyond  the  control  of  the 
decision  maker.  Examples  of  such  events  include  weather, 
political  developments,  and  adversary  dispositions. 

The  second  factor  concerns  the  many  and  diverse  goals 
that  characterize  most  important  decision  problems  and  the 
difficulty  the  decision  maker  has  in  matching  the  multiple 
attributes  of  the  various  decision  outcomes  with  the  multiple 
criteria  for  success. 

Military  decision  problems  of  the  type  described  are 
common,  for  example,  choosing  a  contingency  evacuation  posture 
or  selecting  an  offensive  course  of  action  in  the  face  of 
uncertainty.  Indeed,  the  various  civilian  and  military 
intelligence  agencies  exist  to  assist  the  decision  maker  in 
assessing  the  relative  likelihoods  of  future  events. 

Similarly,  operational  staffs  provide  expertise  in  addressing 
and  focusing  the  various  criteria  for  success. 


There  are  also  many  important  military  decision  prob¬ 
lems  in  which  the  decision-making  process  proceeds  under 
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conditions  of  relative  certainty.  In  those  kinds  of  situa¬ 
tions  it  can  be  assumed  that  the  decision  maker  has,  in 
effect,  complete  information  about  the  alternative  courses 
of  action.  There  are  no  key  uncertainties  to  complicate  the 
decision  outcome.  Decision  problems  of  that  kind  can  be 
represented  graphically  by  a  decision  tree  drawn  in  the 
manner  of  Figure  2-1. 


DECISION 

BLOCK 


DECISION 

ALTERNATIVES 


DECISION 

OUTCOMES 


Figure  2-1 

DECISION  MAKING  UNDER  CERTAINTY 


In  the  figure,  choosing  alternative  leads  inexorably 


to  outcome  0, 


Dj  leads  to  0 2, 


and  so  forth.  The  solution 


is,  in  theory,  deceptively  simple:  choose  the  path  that 
leads  to  the  outcome  providing  the  greatest  satisfaction  to 
the  decision  maker.  In  practice,  however,  the  solution  is 
difficult  because  there  are  multiple  conflictive  criteria 
for  determining  satisfaction.  Inevitably,  there  are  value 
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trade-offs  that  must  be  made,  such  as  performance  versus 
cost  and  risk  versus  yield. 

Examples  of  military  decision  problems  in  an  environ¬ 
ment  of  relative  certainty  include  design-to-price  source 
selection  procedures  in  which  the  source  selection  authority 
must  choose  the  best  item  or  design  from  among  its  equally 
priced  competitors.  The  criteria  comprising  what  the  deci¬ 
sion  maker  believes  constitutes  a  best  design  could  easily 
number  in  the  hundreds. 

Another  example  is  that  of  crisis  decision  making: 
quickly  analyzing  and  choosing  among  several  viable  courses 
of  action  based  on  multiple  criteria  such  as  the  adequacy  of 
forces,  command  and  control  arrangements,  logistics,  sim¬ 
plicity  of  operations,  availability  of  supporting  fires,  and 
so  on.  Evaluation  and  selection  problems  of  all  kinds,  such 
as  personnel  selection,  promotion,  and  the  assignment  of 
missions  to  commands,  all  fall  into  the  classification  of 
decision  making  under  relative  certainty. 

EVAL  is  a  decision-aiding  system  designed  to  help  deci¬ 
sion  makers  solve  problems  of  evaluation  under  relative 
certainty.  EVAL  provides  a  sound  structural  framework  for 
deliberation,  reasoning,  and  analysis.  EVAL  aids  decision 
makers  by  prescribing  a  straightforward  normative  procedure 
for  organizing  and  analyzing  difficult  decision  problems 
involving  complex  value  trade-offs  in  the  choice  of  a  course 
of  action. 

EVAL  has  its  roots  in  decision  analysis,  a  management 
discipline  that  emerged  in  the  1960's.  As  described  in 
reference  1.2.1,  decision  analysis  has  proven  enormously 
effective  in  aiding  military  decision-making  processes 
across  a  broad  spectrum  of  applications.  EVAL,  in  particular. 
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has  been  employed  in  many  source  selection  and  procurement 
decisions  and  in  the  evaluation  of  military  plans. 


2.2  Objective 

EVAL  is  a  decision-analytic  based,  computer-assisted 
evaluation  strategy.  Its  primary  purpose  is  to  provide 
decision  makers  a  procedural  framework  that  will  ensure  that 
their  ultimate  decision  choice  is  a  coherent  one:  a  choice 
fully  consistent  with  their  own  goals  and  value  structures. 

EVAL  provides  a  logical  evaluation  strategy  that  has 
several  very  desirable  characteristics.  First,  EVAL  dis¬ 
criminates  effectively  among  the  various  alternatives  being 
evaluated.  Consistent  with  the  decision  maker's  judgments, 
EVAL  not  only  produces  a  rank  ordering  of  the  alternatives 
in  terms  of  their  overall  utility  to  the  decision  maker,  but 
it  also  scales  the  alternatives  as  well,  indicating  the 
magnitude  of  the  difference  between  any  two  alternatives. 
Second,  EVAL  is  a  reliable  strategy,  in  the  sense  that  the 
results  are  reproducible.  The  specificity  and  visibility  of 
the  model  structure  are  such  that,  almost  without  exception, 
the  same  results  will  be  produced  by  the  same  decision  maker 
working  independently  over  time.  Third,  the  model  facili¬ 
tates  understanding.  Its  structural  framework  provides 
clearly  defined  links  between  the  input  data  and  the  output 
results.  Furthermore,  EVAL  promotes  the  capture  and  preser¬ 
vation  of  the  supporting  rationale  for  all  value  judgments. 
Finally,  the  EVAL  strategy  is  an  equitable  one;  its  visible 
structure  reduces  the  likelihood  of  there  being  inherent 
biases  in  the  evaluation  procedure. 

The  fundamental  product  of  EVAL  is  a  computer- stored 
conceptual  representation,  or  model,  of  the  decision  problem 
at  hand.  Whereas  decision  analysis  provides  the  theoretical 


background  and  procedural  guidance ,  EVAL  provides  the  spe¬ 
cific  methodological  tool  for  processing  information  and 
quantitatively  evaluating  the  various  alternatives  open  to 
the  decision  maker. 

It  must  be  emphasized  that  the  use  of  decision  analysis 
and  EVAL  does  not  replace  human  judgment;  rather,  it  aids 
human  judgment. 

The  overall  objective  of  the  EVAL  system  is  to  provide 
decision  makers  with  the  capability  to  construct,  store,  re¬ 
trieve,  exercise,  and  modify  EVAL  models  of  decision  problems. 
The  user  who  is  inexperienced  in  decision  analysis  is  cautioned 
that  the  EVAL  model  should  not  be  applied  indiscriminately, 
nor  should  its  results  be  interpreted  blindly.  In  particular, 
the  prospective  user  must  understand  that  the  EVAL  framework 
fits  only  those  situations  that  meet  all  of  the  following 
characteristics . 

o  The  evaluation  problem  is  well  formed;  i.e., 

alternative  courses  of  action  have  been  identified. 

o  Multiple  criteria  apply  to  the  evaluation  of  the 
alternative  courses  of  action. 

o  There  is  no  future  key  uncertainty  whose  outcome 
will  impact  the  decision  choice. 

o  A  linear  additive  structural  representation  of  the 
problem  will  suffice. 

o  An  ad  hoc  solution  is  appropriate. 
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2.3  Procedural  Overview 


The  first  step  in  problem  solving  using  EVAL  is  to 
construct  a  conceptual  model  of  the  problem  at  hand.  The 
EVAL  software  is  not  used  during  the  initial  development  of 
the  model;  rather,  the  modeling  process  should  be  a  discovery 
process  that  initially  should  require  several  trial  construc¬ 
tions  using  paper  and  pencil  methods.  The  EVAL  system 
should  be  used  only  when  the  model  has  reached  a  more  ad¬ 
vanced  state  of  refinement. 

The  EVAL  strategy  uses  multi-attribute  utility  (MAU) 
theory  as  the  organizing  framework  for  information  processing. 
MAU  assessment  is  the  methodological  tool  with  which  the 
decision  maker  defines  and  exercises  EVAL  models  to  evaluate 
alternative  systems.  The  EVAL  model  is  used  to  quantify  and 
assess  the  overall  relative  value  of  very  complex  multi- 
attributed  decision  alternatives  that  differ  across  a  large 
number  of  characteristics. 

The  EVAL  model  is  hierarchical  in  nature,  starting  with 
the  overall  top-level  criterion  for  which  a  comparative 
evaluation  score  is  desired.  That  factor  is  successively 
decomposed  into  its  component  criteria  in  descending  levels 
of  the  hierarchy  such  that  each  successive  lower-level  cri¬ 
terion  is  more  specific  than  those  at  the  preceding  level. 

At  the  lowest  level  of  the  hierarchy  are  the  criteria  for 
which  the  user  must  specify  utility  scores  for  each  of  the 
alternatives.  The  criteria  at  the  lowest  level  address  the 
more  easily  understood  operational  or  technical  attributes 
of  the  systems  under  evaluation. 

The  format  of  a  typical  EVAL  MAU  model  is  shown  in 
Figure  2-2.  As  indicated  in  the  figure,  the  decision  maker 
must  decompose  the  overall  evaluation  criterion  into  its 
component  criteria.  Those,  in  turn,  are  decomposed  into 


OVERALL 

EVALUATION  CRITERION 


their  components,  and  so  on  successively  until,  at  the 
bottom  level,  a  criterion  is  so  well  defined  that  there  is 
no  need  for  further  decomposition.  The  bottom-level  cri¬ 
teria  are  circled  in  the  figure. 

Once  the  model  becomes  a  well-structured,  adequate 
representation  of  the  evaluation  criteria,  the  decision 
maker  must  make  two  kinds  of  value  assessments:  relative 
utility  scores  for  the  various  alternatives  and  relative 
importance  weights  for  the  various  criteria. 

For  each  bottom-level  criterion,  the  decision  maker 
must  assess  the  relative  utility  provided  by  each  alterna¬ 
tive.  A  utility  is  a  number  between  zero  and  one  that  rep¬ 
resents  the  relative  degree  of  satisfaction  to  the  decision 
maker  of  an  alternative  with  respect  to  a  particular  bottom- 
level  criterion.  Utilities  are  normally  expressed  as  per¬ 
centages  of  complete  satisfaction.  For  example,  three 
alternative  systems  might  be  scored  50  100  0  with  respect 

to  criterion  2.1.2  in  Figure  2-2.  The  scoring  indicates 
that,  relative  to  only  the  three  alternatives  under  con¬ 
sideration,  the  second  is  the  best,  the  third  is  the  worst, 
and  the  first  falls  midway  between  the  two  with  respect  to 
that  criterion. 

Note  that  utility  scores  are  assessed  only  at  the 
bottom-level  nodes  of  the  structure.  For  that  reason,  the 
bottom-level  criteria  must  be  so  well  defined  that  the 
assignment  of  utility  scores  to  the  alternatives  is  a  rela¬ 
tively  easy  process.  That  is,  the  bottom-level  criteria 
should  foster  the  process  of  discriminating  among  the  alter¬ 
natives. 

Once  the  utility  scores  have  been  assigned  for  the 
bottom-level  criteria,  the  decision  maker  must  assign 
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importance  weights  to  the  criteria.  That  is,  with  the  ex¬ 
ception  of  the  bottom- level  criteria,  each  criterion  in  the 
structure  has  subordinate  components.  The  decision  maker 
must  assess  the  relative  importance  of  the  components  to  the 
aggregate  criterion. 

A  set  of  importance  weights  is  assigned  for  each  decom¬ 
position;  one  weight  for  each  component.  The  weights  them¬ 
selves  represent  percentages  of  the  whole  and  must  sum  to 
100%.  For  example,  the  criteria  comprising  criterion  3.2  in 
Figure  2-2  could  be  assigned  unnormalized  weights  of  50  100  50, 

which  EVAL  would  normalize  to  25  50  25. 

Once  the  model  has  been  completely  structured  and  the 
utility  scores  and  importance  weights  specified,  it  can  be 
exercised  by  the  user  to  produce  the  overall  top-level 
utility  associated  with  each  of  the  decision  alternatives. 

The  rational  user  should  choose  that  alternative  having  the 
greatest  overall  utility. 


The  model  can  be  analyzed  with  respect  to  the  sensi¬ 
tivity  of  the  implied  decision  choice  to  variations  in  the 
weights  assigned  to  the  criteria.  In  addition,  intermediate 
values  of  utility  can  be  obtained  at  any  level  in  the  model. 

2.4  Purpose  of  the  Model 

At  this  point  it  must  be  noted  that  the  purpose  of  an 
EVAL  model  is  not  to  capture  reality,  but  rather  to  approxi¬ 
mate  it.  Structuring  an  evaluation  model  is  an  art,  and  the 
practice  of  that  art  is  attended  by  great  difficulties  in 
selecting  a  representative  set  of  criteria  and  in  structuring 
those  criteria.  Ideally,  an  experienced  professional  decision 
analyst  would  work  closely  with  the  decision  maker  in  struc¬ 
turing  an  EVAL  model.  In  any  case,  the  ultimate  tests  of  an 
EVAL  model  should  be: 


Is  the  model  free  of  obvious  inconsistencies? 

Does  the  model  approximate  the  reality  of  the 
situation? 

Is  the  model  practical  and  useful  to  the  decision 
maker? 

Does  the  model  provide  insight  to  the  decision 
maker  and  the  staff? 


3.0  STRUCTURING  THE  EVAL  MODEL 


In  order  to  use  EVAL,  the  user  must  first  create  an 
EVAL  model.  To  facilitate  understanding  of  the  model  struc' 
turing  process,  this  section  uses  a  hypothetical  case  study 
approach. 


3.1  Hypothetical  Case 


One  of  the  services  is  investigating  the  possibility  of 
consolidating  seven  separate  outmoded  missile  maintenance 
and  repair  facilities  located  throughout  the  continental 
United  States.  None  of  the  current  facilities  are  capable 
of  accommodating  the  combined  workload,  so  that  an  entirely 
new  and  modern  facility  is  required.  Upon  activation  of  the 
new  facility,  the  old  facilities  will  be  deactivated  and 
closed. 


A  preliminary  study  has  uncovered  five  candidate  sites 
for  the  new  facility.  Each  one  of  the  candidates  has  a 
strong  appeal  on  at  least  one  dimension.  The  decision  maker 
is  faced  with  a  dilemma  in  choosing  among  the  five  sites. 
Accordingly,  the  decision  maker  has  decided  to  use  EVAL. 

3.2  The  EVAL  Model 

The  first  step  in  using  EVAL  is  to  structure  an  EVAL 
model.  The  model  is  structured  criterion-by-criterion  from 
the  top  down  by  successively  decomposing  each  criterion  in 
turn  until,  at  the  bottom  level,  a  criterion  is  so  well 
defined  that  there  is  no  need  for  further  decomposition. 

The  format  of  the  model  should  resemble  the  general  format 
shown  in  Figure  2-2. 

Initially  the  structure  of  the  model  will  require 
several  iterations  and  refinements.  For  that  reason,  the 


first  stages  of  structuring  should  be  done  using  paper  and 
pencil  rather  than  with  the  EVAL  software. 

When  the  model  structure  is  well  formed,  the  user 
creates  an  EVAL  model.  A  complete  EVAL  model  consists  of 
the  following  elements. 

3.2.1  The  nature  of  the  evaluation  problem  -  A  short 
label  defining  the  problem.  This  label  distinguishes  the 
EVAL  model  as  a  whole  and  is  used  by  the  system  to  store  and 
retrieve  the  model.  In  the  case  at  hand,  SITES  is  an  appro¬ 
priate  label. 

3.2.2  The  systems  -  A  list  of  the  various  systems  to 
be  evaluated.  The  term  "systems"  is  used  in  the  generic 
sense;  it  could  refer  to  items  to  be  procured,  fiscal  pro¬ 
grams  to  be  funded,  personnel  to  be  evaluated,  and  so  on. 

In  the  site  selection  problem  the  systems  to  be  evaluated 
are  the  five  sites.  They  will  be  referred  to  hereafter  as 
site  A,  site  B,  site  C,  site  D,  and  site  E. 

3.2.3  The  criteria  -  A  hierarchical  set  of  criteria 
constituting  the  overall  evaluation.  The  top-level  evalua¬ 
tion  criterion  is  decomposed  into  its  immediate  component 
criteria  which  themselves  may  be  further  decomposed.  Those 
criteria  which  are  not  further  decomposed  constitute  the 
bottom-level  criteria. 

Each  criterion  is  assigned  a  data  identification 
number  in  the  manner  shown  in  Figure  2-2.  The  number  de¬ 
scribes  how  all  of  the  various  criteria  are  logically  re¬ 
lated.  For  example,  criterion  4  would  decompose  into 
criteria  4.1,  4.2,...,4.n.  Similarly,  criterion  4.4  would 
decompose  into  criteria  4.4.1,  4.4.2, ... ,4.4. m.  The  top- 
level  criterion  is  usually  assigned  the  data  identification 
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number  0  and  its  immediately  successive  criteria  assigned 
the  numbers  1,  2,  3... 

Figure  3-1  is  a  representative/  although  incom¬ 
plete/  structure  of  the  criteria  relevant  to  the  site  selec¬ 
tion  problem.  Only  two  bottom- level  criteria  are  shown  in 
the  figure;  both  are  circled.  The  figure  is  notional;  other 
criteria  could  be  added. 

The  user  should  provide  concise  written  defini¬ 
tions  of  the  criteria.  Bottom-level  criteria/  in  particular, 
must  be  defined  precisely  since  they  are  the  only  criteria 
for  which  the  alternatives  will  be  scored  with  respect  to 
one  another.  For  example,  criterion  1.3. 3.1.1  might  be 
defined  as  "...the  total  amount  of  existing  covered  ware¬ 
house  storage  space  including  open-sided  shed  storage." 

3.2.4  Utility  scores  -  For  each  bottom- level  cri¬ 
terion,  the  user  must  assess  the  relative  utility  of  each 
alternative.  Utilities  must  range  in  value  from  100  for 
that  alternative  providing  the  highest  degree  of  satisfac¬ 
tion  to  0  for  that  alternative  providing  the  least  satis¬ 
faction.  Appropriate  intermediate  values  of  utility  are 
assigned  to  the  remaining  alternatives.  Written  rationale 
should  support  the  utility  assessments. 

It  is  important  to  note  that  the  alternatives 
must  be  assessed  across  the  full  range  of  value  of  utility: 

0  to  100,  inclusive.  The  overall  importance  of  the  differ¬ 
ence  between  the  worst  (0)  and  best  (100)  alternatives  will 
be  assessed  later,  but  at  this  point  both  values  (0  and  100) 
must  be  assigned.  The  purpose  of  the  model  is  to  discrimi¬ 
nate  among  the  specified  alternatives.  That  discrimination 
must  be  made  at  each  bottom- level  criterion. 
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For  the  site  selection  case,  a  representative 
assessment  of  utilities  is  shown  in  Figure  3-2.  Note  that  a 
particular  value  of  utility  may  be  assigned  to  more  than  one 
alternative.  Again,  the  user  is  reminded  that  written 
rationale  should  support  each  utility  assessment. 

Criterion  1.3. 3.1.2 

OVERALL— SUPPORT  OF  MISSION— EXISTING 
SPACE— AVAILABLE  STORAGE— COVERED— QUALITY 


SITE  A 

- 

25 

SITE  B 

- 

0 

SITE  C 

- 

100 

SITE  D 

- 

25 

SITE  E 

- 

90 

Figure  3-2 

REPRESENTATIVE  UTILITY  VALUES 

3.2.5  Criteria  importance  weights  -  Proceeding  from 
the  bottom-level  criteria  to  the  top,  the  user  must  assess 


the  relative  importance  of  the  decomposed  criteria.  For 
example,  in  the  site  selection  problem  the  user  must  deter¬ 
mine  the  relative  importance  of  the  quality  versus  the 
quantity  of  available  covered  storage.  That  determination 
should  not  be  based  on  some  global,  absolute  notion.  Rather 
the  determination  must  be  based  on  the  particular  attributes 
of  the  sites  under  consideration.  It  must  be  based  on  the 
relative  difference  between  the  best  and  worst  sites  regard¬ 
ing  quality  and  quantity.  For  example,  Figure  3-2  shows 
that  regarding  quality,  site  B  is  worst  and  site  C  is  best. 
Assume  that  regarding  quantity,  site  A  is  worst  and  site  E 
is  best.  In  assessing  the  relative  importance  of  quality 
and  quantity  the  user  must  mentally  substitute  for  quality 


the  difference  between  sites  B  and  C  and  for  quantity  the 
difference  between  sites  A  and  E.  Assume  that  the  relative 
difference  in  quality  is  judged  four  times  more  important 
than  the  relative  difference  in  quantity.  Then  the  most 
important  of  those  two  criteria,  quality,  would  be  assigned 
a  weight  of  100  versus  a  weight  of  25  assigned  to  quantity. 
EVAL  would  normalize  those  weights  to  80,  20  so  that  they 
sum  to  100%. 

The  user  proceeds  in  that  fashion,  bottom  to 
top,  comparing  differences  until  each  criterion  has  been 
assessed.  As  the  user  proceeds  toward  the  top,  each  crite¬ 
rion  is  evaluated  by  using  its  most  important  component  as 
a  surrogate.  For  example,  using  Figure  3-1,  in  assessing 
the  relative  importance  of  covered  versus  uncovered  storage, 
the  user  compares  the  difference  in  the  quality  of  covered 
storage  with  the  difference  in  the  most  important  factor 
comprising  uncovered  storage.  Only  the  bottom-level  differ¬ 
ences  are  used  in  the  weighting  procedure,  each  being  used 
to  represent  a  higher  level  criterion. 

In  that  manner,  relative  importance  weights  are 
assigned  to  each  criterion  until,  finally,  weights  are 
assessed  for  the  relative  importance  of  criterion  1,  support 
of  mission;  criterion  2,  support  of  relocation;  and  crite¬ 
rion  3,  political  consideration.  As  before,  those  weights 
are  assessed  by  comparing  the  difference  existing  between 
the  most  important  bottom- level  criterion  in  each  area. 

For  example,  consider  the  structure  shown  in 
Figure  3-3.  Assume  that  the  decision  maker  judges  bottom- 
level  criterion  1.2.1  more  important  than  1.2.2,  and  cri¬ 
terion  2.1  more  important  than  2.2  and  2.3.  The  more  impor¬ 
tant  criteria  (1.2.1  and  2.1)  are  shaded  in  the  figure.  The 
next  step  is  to  compare  criterion  1.1  with  1.2.  At  that 
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Figure  3-3 

A  SAMPLE  STRUCTURE 


level,  criterion  1.2.1  is  used  to  represent  1.2.  The 
required  comparison  is  between  criteria  1.1  and  1.2.1. 
Again,  assume  that  criterion  1.2.1  is  judged  most  important 
Criterion  1.2.1  now  becomes  a  surrogate  for  criterion  1, 
where  it  is  compared  with  criterion  2.1,  the  surrogate  for 
criterion  2. 


It  is  always  the  bottom-level  criteria  that  are 
compared  in  the  process  of  determining  importance  weights. 
In  that  manner,  proceeding  from  bottom  to  top,  weights  are 
established  for  all  of  the  criteria  in  the  model.  The 
relative  importance  weights  might  be  as  shown  in  Figure  3-4 


Figure  34 

IMPORTANCE  WEIGHTS 


An  EVAL  model  has  been  completely  structured 
when:  (1)  the  systems  to  be  evaluated  have  been  identified; 

(2)  the  evaluation  criteria  are  identified,  defined,  and 
interrelated;  (3)  the  utility  scores  for  each  alternative 
are  assessed  at  the  bottom-level  criteria  and  the  supporting 
rationale  for  the  scores  is  captured;  and  (4)  the  importance 
weights  are  assessed  for  each  criterion  and  the  supporting 
rationale  for  the  weights  is  captured. 


4.0  RESULTS  OF  THE  MODEL 


» 

Once  an  EVAL  model  has  been  completely  specified  it  can 
be  processed  to  produce  the  following  results  of  interest  to 
the  user. 

i 

4.1  Overall  Result 

EVAL  will  display  the  overall  utility  associated  with 
each  alternative.  For  each  alternative  the  overall  utility 
is  computed  by  properly  weighting  and  adding  together  the 
utility  scores,  beginning  with  the  bottom-level  criteria  and 
so  continuing  to  the  top  of  the  hierarchy.  The  utility 
associated  with  any  one  criterion  (except  those  at  the 
bottom- level)  is  the  weighted  sum  of  the  utilities  at  the 
next  lower  level.  EVAL  thus  implements  a  linear  additive 
evaluation  procedure. 

For  example,  in  the  site  selection  problem  discussed  in 
Section  3,  the  overall  result  might  be  as  shown  in  Figure  4-1. 
The  figure  displays  the  utility  values  of  the  five  sites  for 
each  of  the  successive  (next  lower  level)  components  as  well 
as  each  component's  relative  importance  weight  (shown  in 
parentheses) .  The  overall  utility  of  each  site  is  shown 
under  the  heading  "TOTAL. "  Consistent  with  the  decision 
maker's  values  and  importance  weights,  site  D  should  be 
selected  for  the  new  facility. 

4.2  Intermediate  Results 

EVAL  can  display  results  similar  to  those  shown  in 
Figure  4-1  for  any  desired  criterion  other  than  those  at  the 
bottom  level.  Bottom-level  criteria  weights  and  utility 
score 8  can  be  observed  by  displaying  the  next  higher  level 
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SITE 
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A 

B 

C 

D 

E 

1. 
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(52) 

61 

72 

45 

80 

75 

2. 

SUPPORT  OF  RELOCATION 

(33) 

20 

62 

70 

61 

38 

3. 

POLITICAL  CONSIDERA¬ 
TIONS 

(15) 

81 

43 

52 

76 

65 
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50 

64 

54 

73 

61 

Figure  4-1 

SAMPLE  OVERALL  RESULTS 


criterion.  For  example,  the  two  bottom- level  criteria  shown 
in  Figure  3-1  can  be  observed  by  displaying  criterion  1.3. 3.1, 
as  shown  in  Figure  4-2.  Note  that  cumulative  weights  are 
shown. 


Criterion  1.3. 3.1 

SUPPORT  OF  MISSION— EXISTING  SPACE— AVAILABLE 
STORAGE— COVERED 


SITE 

A 

SITE 

B 

SITE 

C 

SITE 

D 

SITE 

E 

CUMWT 

1.3. 3. 1.1 

QUANTITY 

(80) 

0 

33 

67 

100 

75 

15 

1.3. 3. 1.2 

QUALITY 

(20) 

25 

0 

100 

25 

90 

4 

TOTAL 

5 

26 

74 

85 

78 

19 

Figure  4-2 

INTERMEDIATE  RESULTS 
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4.3  Sensitivity  Analysis 


Variations  in  the  cumulative  weights  assigned  to  the 
various  criteria  will  produce  corresponding  variations  in 
the  overall  results.  The  user  should  be  interested  in  how 
sensitive  the  final  results  are  to  such  variations.  EVAL 
will  perform  such  a  sensitivity  analysis. 

The  user  identifies  any  desired  criterion.  EVAL  then 
displays  the  current  cumulative  weight  of  that  criterion  and 
asks  the  user  to  designate  minimum  and  maximum  test  values 
of  cumulative  weight  for  that  criterion.  EVAL  then  varies 
the  cumulative  weight  from  minimum  to  maximum  in  ten  equal 
increments,  keeping  the  weights  of  the  other  criteria  in 
their  proper  proportion. 

A  sample  analysis  that  might  be  performed  in  the  site 
selection  problem  is  shown  in  Figure  4-3.  Note  that  the 
utility  scores  that  are  shown  represent  the  overall  (highest 
level)  utility.  The  highest  utility  score  is  identified 
with  an  asterisk.  The  analysis  shows  that  the  overall 
result  is  indeed  sensitive  to  the  weight  assigned  to  the 
quantity  of  covered  storage.  As  that  criterion  becomes  more 
important  as  its  cumulative  weight  is  varied  from  0  to  50, 
site  B  becomes  preferable  to  site  D. 
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Criterion  1.3. 3. 1.1 

SUPPORT  OF  MISSION— EXISTING  SPACE— AVAILABLE 
STORAGE— COVE  RED-- QUALI TY 
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SAMPLE  SENSITIVITY  ANALYSIS 
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5.0  TECHNICAL  OPERATIONS 


EVAL  consists  of  two  subsystems :  STRUCTURE  and  RUN. 

The  STRUCTURE  subsystem  is  used  to  construct  a  new  evalua¬ 
tion  model  or  to  edit  the  structure  of  an  existing  model. 

Its  function  is  solely  to  specify  and  relate  the  hierarchical 
criteria;  it  does  not  elicit  utility  scores  or  importance 
weights. 

Utility  scores  and  importance  weights  are  entered  by 
using  the  RUN  subsystem.  RUN  is  also  used  to  edit  previously 
entered  scores  and  weights,  and  it  is  used  to  display  the 
model  results  as  described  in  Section  4.0. 


5.1  Option  Menus 

Both  subsystems  are  hierarchically  structured  and  menu- 
driven.  The  user  initiates  the  various  subsystem  functions 
by  progressing  through  a  hierarchy  of  menus,  each  menu  pre¬ 
senting  several  functions  to  the  user.  In  general,  selecting 
a  function  either  causes  a  more  detailed  menu  to  be  displayed 
or  causes  the  function  to  be  executed.  Not  choosing  any  of 
the  menu  functions  causes  the  previous  menu  to  be  displayed. 


5.2  The  STRUCTURE  Subsystem 


5.2.1  The  primary  menu  -  After  the  user  has  loaded  the 
STRUCTURE  subsystem  into  the  computer,  a  primary  menu  will 
be  displayed.  This  menu  contains  eight  functions: 


o  Load  Model 
o  Edit  Structure 
o  Create  or  Add  to  a  Structure 
o  Save  Model 


o  Develop  Structure 
o  Create  Branch 
o  Prune  Section 
o  Print  Review  Sheet. 

In  order  to  use  the  STRUCTURE  software,  the  user 
must  first  select  either  the  "Load  Model"  function  to  load 
an  already  existing  model  or  the  "Create  or  Add  to  a  Struc¬ 
ture"  or  "Create  Branch"  functions  to  create  a  brand  new 
model. 

Load  Model  -  Selecting  "Load  Model"  will  cause 
the  subsystem  to  display  the  names  of  the  existing  models 
that  are  available  for  loading  from  the  model  library  and 
will  instruct  the  user  on  how  to  load  the  model  into  the 
computer.  Once  the  user  has  loaded  the  desired  model  into 
the  computer,  the  model  is  available  for  editing  (the  "Edit 
Structure"  option) ,  expanding  (the  "Create  or  Add  to  a 
Structure"  option) ,  contracting  (the  "Prune  Section"  option) , 
or  reviewing  (the  "Print  Review  Sheet"  option) . 

Edit  Structure  -  This  function  allows  the  user 
to  edit  the  name  or  data  identification  number  (DIN)  of  any 
criterion  in  the  model  structure.  Also,  any  desired  cri¬ 
terion  may  be  deleted  from  the  model. 

When  "Edit  Structure"  is  selected,  the  computer 
responds  by  requesting  the  DIN  of  the  desired  criterion. 

When  that  is  specified,  the  computer  displays  the  DIN  and 
its  associated  label.  The  user  then  either  makes  whatever 
changes  are  desired  or,  by  returning  the  carriage  without 
entering  a  change,  the  user  may  delete  the  criterion  entirely 

Create  or  Add  to  a  Structure  -  This  function  is 
used  either  to  create  a  new  model  structure  or  to  add  to  an 


•xi sting  model  which  has  been  loaded  using  the  "Load  Model” 
function.  The  user  is  first  asked  whether  the  model  is  a 
brand  new  one.  If  it  is,  the  computer  will  request  the 
names  of  the  various  systems  to  be  evaluated.  If  not,  the 
previously  loaded  model  is  used.  In  either  case,  the  fol¬ 
lowing  procedure  is  followed. 

The  computer  prompts  the  user,  requesting  the 
DIN  and  label  of  whatever  criterion  node  the  user  wishes  to 
enter.  The  user  types  both  items  on  the  same  line,  separated 
by  commas.  The  criterion  numbering  scheme  is  optional,  but 
it  is  recommended  that  the  user  follow  the  scheme  shown  in 
Figure  3-1.  Any  desired  name  may  be  assigned  to  a  criterion, 
provided  that  the  name  does  not  contain  any  commas.  After 
each  criterion  is  entered,  the  computer  will  display  it  for 
the  user.  If  an  error  is  detected,  the  user  may  simply 
retype  the  name.  However,  if  the  DIN  is  wrong,  it  can  only 
be  corrected  by  using  "Edit  Structure." 

Instead  of  responding  to  the  computer's  request 
by  entering  a  new  criterion,  the  user  may  instead  attach  an 
entire  branch  that  was  previously  created  by  using  the 
"Create  Branch"  function.  This  is  done  by  typing  the  branch 
designation  character  followed  by  the  branch  number  and  the 
number  of  the  DIN  to  which  the  branch  should  be  attached. 


Once  all  criterion  nodes  and  branches  have  been 
attached,  the  user  can  return  to  the  primary  menu  by  returning 
the  carriage  without  responding  to  the  computer's  prompt. 

Develop  Structure  -  In  order  for  a  model  to  be 
stored  permanently  in  the  model  library,  the  model's  struc¬ 
ture  must  first  be  organized  into  a  particular  format  for 
internal  use.  The  "Develop  Structure"  option  accomplishes 


this.  When  this  option  is  selected,  the  computer  will  dis¬ 
play  an  estimate  of  the  amount  of  time  necessary  to  develop 
the  structure.  When  the  development  has  been  completed,  the 
computer  will  inform  the  user  that  it  is  finished.  The 
model  can  now  be  stored  using  the  "Save  Model"  function. 

Save  Model  -  The  "Save  Model”  function  allows 
the  user  to  store  a  newly  created  or  edited  model  in  the 
computer's  library.  When  the  user  selects  this  function, 
the  computer  will  respond  by  asking  the  user  to  confirm  that 
the  structure  has  been  developed  using  the  "Develop  Struc¬ 
ture"  function.  A  negative  response  will  cause  the  computer 
to  develop  the  structure  before  proceeding.  The  computer 
will  list  the  names  of  the  models  already  stored  in  the 
model  library.  The  user  is  then  asked  for  the  name  of  the 
model  to  be  saved.  If  the  name  proposed  by  the  user  duplicates 
the  name  of  an  existing  model,  the  computer  will  request 
confirmation  that  the  new  model  should  replace  the  old  one 
of  the  same  name.  If  the  name  is  a  new  one,  its  name  is 
added  to  the  list  of  models  available  and  it  is  saved  per¬ 
manently  . 

Create  Branch  -  This  function  is  used  to  create 
repetitive  portions  of  the  model  by  specifying  a  single 
branch.  That  branch  has  its  own  hierarchical  organization 
and  can  be  added  to  the  overall  structure  at  any  place  it  is 
needed.  The  user  can  save  a  significant  amount  of  time  and 
effort  by  employing  the  "Create  Branch"  function. 

When  this  function  is  selected,  the  computer 
will  request  the  Branch  Identification  Number.  This  will  be 
needed  later  when  the  branch  is  added  to  the  overall  struc¬ 
ture.  Each  separate  branch  must  be  assigned  a  unique  integral 
Branch  Identification  Number.  The  computer  will  then  ask 
whether  the  branch  is  a  new  branch.  The  user  responds 


positively  or  negatively  according  to  whether  an  entirely 
new  branch  is  being  created  or  an  existing  branch  is  being 
edited. 

The  computer  then  asks  if  the  format  of  the 
branch  is  symmetric.  If  the  branch  is  symmetric,  the  com¬ 
puter  simplifies  the  input  elicitation  by  allowing  the  user 
to  input  at  one  time  the  names  of  all  the  criterion  nodes 
appearing  repetitively  at  each  level.  This  is  done  by 
specifying  the  number  of  levels  in  the  branch.  The  names  of 
the  first-level  criteria  are  then  typed  in,  one  per  line, 
returning  the  carriage  between  nodes.  When  the  carriage  is 
returned  without  typing  anything,  the  computer  asks  for  the 
criterion  names  at  the  next  level.  This  process  continues 
until  all  levels  have  been  completed.  The  computer  will 
then  ask  for  a  new  Branch  Identification  Number. 

If  the  branch  is  asymmetric,  the  computer  will 
prompt  the  user  to  type  in  each  branch  DIN  and  the  criterion 
node  name.  This  process  continues  until  the  user  returns 
the  carriage  without  responding.  The  computer  will  then 
request  the  next  Branch  Identification  Number. 

Prune  Section  -  This  function  allows  the  user  to 
delete  from  a  structure  all  nodes  emitting  from  a  specified 
criterion  node.  This  is  required  if  a  section  of  the  tree 
proves  inaccurate  or  unnecessary. 

When  this  function  is  selected,  the  user  is 
asked  to  specify  the  DIN  of  the  node  at  which  all  lower- 
level  nodes  are  to  be  deleted.  When  the  user  specifies  the 
DIN,  the  computer  displays  the  name  and  DIN  of  the  node  and 
requests  confirmation  that  all  of  the  attached  nodes  should 
be  deleted.  Once  given  confirmation,  the  computer  deletes 
the  appropriate  nodes  and  informs  the  user  of  how  many  nodes 


were  deleted.  It  then  asks  for  the  DIN  of  the  next  dele¬ 
tion.  If  no  additional  deletion  is  required,  the  user 
simply  returns  the  carriage  without  specifying  a  DIN.  Note 
that  the  specific  node  identified  with  the  selected  DIN  is 
not  itself  deleted — only  attached  nodes  are  removed. 

Print  Review  Sheet  -  This  function  provides  the 
user  with  a  printed  record  of  the  nodes  (DIN's  and  criterion 
names)  in  the  model.  This  display  should  be  obtained  before 
developing  and  saving  the  model  to  confirm  that  the  model 
has  been  properly  constructed.  When  the  "Print  Review 
Sheet"  function  is  selected,  the  computer  will  instruct  the 
user  as  to  how  to  obtain  the  printed  copy. 

5.3  The  RUN  Subsystem 

5.3.1  The  primary  menu  -  After  the  user  had  loaded  the 
RUN  program  into  the  computer,  the  computer  will  explain  how 
to  load  a  model  from  the  model  library.  Once  a  desired 
model  is  loaded,  the  primary  menu  is  displayed.  This  menu 
contains  eight  functions: 

o  Display  Results 
o  Sensitivity 
o  Edit  Values 
o  Print  Results 
o  Load  Model 
o  Save  Model 
o  New  Values 
o  Print  Data  Sheet. 

Each  of  these  functions  is  discussed  in  the 


sections  which  follow. 


Display  Result*  -  This  function  allows  the  user 
to  examine  the  evaluation  results.  By  specifying  a  DIN,  the 
user  is  presented  with  a  matrix  showing  the  total  utility  at 
that  level  of  the  model  for  each  system  being  evaluated. 

The  display  also  shows  the  utilities  at  the  next-lower 
level,  the  relative  importance  weights  of  the  criteria  and 
their  overall  cumulative  weights. 

When  the  "Display  Results"  option  is  selected, 
the  computer  will  request  the  DIN  of  the  node  of  interest  to 
the  user.  Typing  in  the  appropriate  DIN  will  produce  the 
display  described  above,  followed  by  a  request  for  the  DIN 
of  the  next  node  of  interest.  Returning  the  carriage  with¬ 
out  entering  a  DIN  will  return  the  user  to  the  primary  menu. 

If  the  user  types  the  DIN  of  a  node  which  does 
not  exist,  the  computer  will  display  a  message  with  that 
information.  If  the  user  types  the  DIN  of  a  bottom-level 
node,  the  computer  will  display  a  message  informing  the  user 
that  the  requested  node  is  a  bottom- level  node  which  cannot 
be  displayed  because  it  has  no  nodes  emitting  from  it.  To 
view  a  bottom- level  node,  the  user  must  select  the  DIN  of 
the  node  to  which  it  is  attached. 

Sensitivity  -  The  "Sensitivity"  function  permits 
the  user  to  examine  the  effect  on  the  overall  results  of  the 
model  caused  by  changing  the  weight  of  any  particular  cri¬ 
terion  in  the  structure  without  having  to  actually  edit 
the  previously  existing  weight.  It  does  this  by  varying 
the  cumulative  weight  (the  percentage  contribution  of  that 
particular  criterion  to  the  overall  result)  over  a  range  of 
values  selected  by  the  user. 

When  the  "Sensitivity"  func  .  .on  is  selected, 
the  computer  requests  the  DIN  of  the  criterion  to  be  examined 


Once  that  is  specified ,  the  computer  informs  the  user  of  the 
criterion's  current  cumulative  weight  and  requests  the  user 
to  specify  the  minimum  and  maximum  weights  of  interest. 

Once  those  are  specified/  the  computer  will  supply  the 
overall  utility  scores  for  each  of  the  systems  under  evalua¬ 
tion  in  ten  intervals  between  the  minimum  and  maximum  weights. 
Altogether/  eleven  utility  scores  will  appear  for  each 
system,  and  the  system  with  the  highest  utility  score  for 
each  posited  cumulative  weight  will  be  designated  with  an 
asterisk.  Returning  the  carriage  will  allow  the  user  to 
designate  another  DIN  for  examination. 

Edit  Values  -  The  "Edit  Values"  function  allows 
the  user  to  change  either  the  utility  scores  or  the  criterion 
weights.  When  this  function  is  selected,  the  user  is  asked 
whether  it  is  weights  or  scores  that  require  editing. 

If  "Edit  Weights"  is  selected,  the  user  is  asked 
to  supply  the  DIN  of  the  criterion  to  which  are  attached  the 
lower-level  criteria  to  be  reweighted.  The  old  weights  will 
be  shown  and  the  new  weights  requested.  The  user  may  now 
designate  new  weights  and  the  computer  will  automatically 
normalize  them  to  sum  to  100%.  The  computer  requests  con¬ 
firmation  of  the  normalized  weights  before  requesting  the 
next  DIN. 

If  "Edit  Scores"  is  selected,  the  user  is  asked 
to  supply  the  DIN  of  the  bottom- level  criterion  where  the 
scores  are  to  be  edited.  The  existing  scores  will  then  be 
displayed  and  the  user  will  be  allowed  to  edit  them.  If  the 
DIN  designated  by  the  user  is  not  a  data  input  node,  the 
computer  will  display  a  message  informing  the  user  thereof. 

Print  Results  -  The  "Print  Results"  function 
permits  the  user  to  obtain  a  complete  printout  of  the 


results  for  all  of  the  criteria  in  the  model  as  a  permanent 
record  of  the  evaluation.  It  is  also  useful  as  a  means  for 
reviewing  the  results  of  a  large  model  containing  many 
criteria,  rather  than  utilizing  the  more  cumbersome  and  time 
consuming  "Display  Results"  function. 

When  "Print  Results"  is  selected,  the  computer 
will  inform  the  user  how  to  obtain  a  printed  output  from  the 
computer  system  on  which  EVAL  is  implemented.  The  computer 
will  then  allow  the  user  to  type  in  an  appropriate  and 
descriptive  title  for  the  display.  The  computer  produces  a 
listing  of  all  of  the  criteria  matrices  in  numerical  order 
by  DIN. 


Load  Model  -  This  function  allows  the  user  to 
select  a  previously  constructed  model  from  the  model  library 
and  load  it  into  the  computer  workspace  for  purposes  of 
display  or  editing.  When  the  RUN  subsystem  software  is 
loaded,  the  computer  automatically  selects  the  "Load  Model" 
option  because  nothing  constructive  can  be  accomplished  in 
EVAL  before  a  model  is  loaded.  However,  if  a  user  has 
finished  displaying  and  editing  one  model,  the  "Load  Model" 
function  may  be  selected  to  allow  the  user  to  load  a  second 
model  without  having  to  reload  the  whole  RUN  subsystem. 

Save  Model  -  This  function  allows  the  user  to 
store  the  current  model  permanently  in  the  model  library. 

If  an  existing  model  has  been  loaded  and  displayed  but  no 
changes  have  been  made,  it  is  not  necessary  to  save  the 
model. 


When  the  "Save  Model"  function  is  selected,  the 
computer  will  list  the  names  of  the  models  which  are  cur¬ 
rently  stored  in  the  model  library.  The  user  is  then  asked 
for  the  name  of  the  model  to  be  saved.  If  the  name  input  by 
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the  user  is  identical  to  that  of  an  existing  model,  the 
computer  will  request  confirmation  that  substitution  of  the 
new  version  for  the  old  is  desired  before  it  deletes  the  old 
version  and  saves  the  new  one.  If  the  name  entered  is  a  new 
one,  the  computer  saves  the  model  and  adds  the  model's  name 
to  the  list  of  available  models. 

New  Values  -  This  function  is  used  to  enter  a 
complete  set  of  weights  and/or  scores  into  the  model.  This 
is  normally  done  only  with  a  newly  created  model,  since  the 
"Edit  Values"  option  is  available  to  make  individual  changes 
in  weights  and  scores. 

When  this  function  is  selected,  the  user  will  be 
presented  with  another  menu  requesting  that  either  "Load 
Scores"  or  "Load  Weights"  be  selected.  Selecting  "Load 
Scores"  causes  the  computer  to  print  out  the  criteria  line- 
by-line  (in  order  by  DIN) ,  awaiting  user  input  of  scores 
whenever  a  bottom-level  criterion  is  reached.  Above  this 
list  appears  the  names  of  the  systems  being  evaluated.  The 
user  must  type  in  the  utility  scores  for  each  system  under 
evaluation,  leaving  spaces  between  the  numbers.  This  pro¬ 
cess  is  continued  until  the  final  criterion  is  reached. 

When  "Load  Weights"  is  selected,  the  computer 
will  print  out  the  current  importance  weights  assigned  to 
the  criteria.  Those  will  be  all  zeroes  for  a  new  model. 

The  user  must  input  the  correct  weights  assigned  to  the 
various  criteria  comprising  each  node.  Once  the  relative 
weights  have  been  input,  the  computer  normalizes  them  to  sum 
to  100%  and  requests  confirmation  of  the  result.  Once 
confirmation  is  given,  the  computer  proceeds  to  the  next 
node . 
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Print  Data  Sheet  -  The  "Print  Data  Sheet"  func¬ 
tion  allows  the  user  to  obtain  a  printout  of  the  entire 
model  for  use  in  recording  and  entering  weights  and  scores 
which  are  not  available  or  finalized  when  the  model  is  first 
created.  It  acts  as  a  convenient  recording  device  when  an 
individual  or  group  is  arriving  at  the  weights  and  scores. 

It  clearly  displays  the  entire  model  structure  so  that  the 
user  can  note  each  criterion's  relationship  to  the  whole. 

When  this  function  is  selected,  the  computer 
will  inform  the  user  how  to  obtain  a  printed  output  from  the 
computer  system  on  which  EVAL  is  implemented. 
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6.0  AN  EXAMPLE  OF  THE  USE  OF  EVAL 


This  section  presents  an  example  evaluation  procedure 
conducted  using  the  EVAL  system.  The  example  is  based  on  a 
hypothetical  problem  which  involves  choosing  one  of  five 
candidate  aircraft.  The  evaluation  is  based  on  more  than 
thirty  distinct  criteria. 


6 . 1  Hypothetical  Problem 


A  DoD  agency  is  investigating  purchasing  an  executive 
aircraft  in  order  to  promote  cost-effective  and  timely 
travel  by  its  senior  executive  staff,  both  military  and 
civilian.  The  agency  has  narrowed  the  choice  of  aircraft  to 
five,  which  will  hereafter  be  referred  to  as  aircraft  A,  B, 
C,  D,  and  E.  The  director  of  the  agency  has  identified 
three  major  (highest  level)  selection  criteria:  cost, 
performance,  and  miscellaneous  factors. 

Each  of  the  five  aircraft  is  very  attractive  on  one  or 
two  of  the  criteria;  no  single  aircraft  provides  maximum 
benefit  on  all  three  criteria.  Faced  with  a  very  difficult 
choice,  the  director  has  decided  to  use  EVAL  as  a  decision 
aid. 


6.2  The  Approach 

The  first  step  in  using  EVAL  is  to  formulate  a  hier¬ 
archical  evaluation  structure:  to  add  specificity  by  decom¬ 
posing  the  three  highest  level  criteria  into  their  components. 
This  step  should  be  performed  through  normal  working  pro¬ 
cedures;  it  does  not  require  the  use  of  the  EVAL  software. 

Assume  that  after  several  hours,  agreement  was  reached 
on  the  structural  format  shown  in  Figure  6-1.  Note  that  the 
three  highest  level  criteria  have  been  successively  decom¬ 
posed  to  produce  twenty- seven  bottom- level  criteria.  - 
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OVERALL 


Figure  6-1 

EVALUATION  STRUCTURE 


Having  reached  initial  agreement  on  the  structure,  the 
user  is  now  ready  to  use  the  EVAL  software  system.  Since  a 
brand  new  model  is  to  be  created,  the  user  begins  by  loading 
the  STRUCTURE  subsystem  of  EVAL  into  the  computer. 

The  remainder  of  the  figures  in  this  section  include 
representative  user  inputs  and  corresponding  system  outputs. 
Other  input  formats  would  be  equally  suitable. 

6 . 3  Using  the  STRUCTURE  Software  to  Structure  the  Example 
Problem 


Having  loaded  the  STRUCTURE  software,  the  user  will 
observe  a  menu  of  options  such  as  the  one  shown  in  Figure 
6-2.  In  this  and  all  succeeding  figures,  user  inputs  are 
underlined  for  clarity. 


SELECT  THE  NUMBER  OF  THE  OPTION  YOU  DESIRE 

1)  LOAD  MODEL 

2)  EDIT  STRUCTURE 

3)  CREATE  OR  ADD  TO  A  STRUCTURE 

4)  SAVE  MODEL 

5)  DEVELOP  STRUCTURE 

6)  CREATE  BRANCH 

7)  PRUNE  SECTION 

8)  PRINT  REVIEW  SHEET 
SELECTION!  6 


Figure  6-2 

STRUCTURE — THE  PRIMARY  MENU 


For  illustrative  purposes,  assume  that  the  user  has 
decided  to  construct  the  evaluation  model  in  two  stages: 
first,  by  creating  two  separate  branches — one  extending  from 
criterion  1.1  (COST)  and  one  extending  from  criterion  1.3 
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(MISC) ;  and  second,  by  creating  the  remainder  of  the  struc¬ 
ture  and  then  adding  the  branches.  Branches  are  created  by 
selecting  the  "Create  Branch"  function. 

When  "Create  Branch"  is  selected,  the  computer  requests 
the  branch  identification  number  and  then  asks  what  type  of 
branch,  symmetric  or  asymmetric,  is  to  be  constructed.  Note 
that  in  Figure  6-1,  the  COST  branch  is  asymmetric  and  the 
MISC  branch  is  symmetric,  the  symmetry  referring  to  the 
repetition  of  the  lower- level  criteria.  The  exchange  takes 
place  as  shown  in  Figure  6-3  for  the  COST  branch.  Note  that 
the  branch  identification  number  is  arbitrary. 


ENTER  THE  BRANCH  IDENTIFICATION  NUMBER:  1 
IS  THIS  A  NEW  BRANCH?  YES 
IS  THIS  A  SYMMETRIC  BRAISE?  NO 


Figure  6-3 

DEFINING  THE  BRANCH  NUMBER  AND  TYPE 


The  user  will  now  be  asked  to  enter  the  branch  cri¬ 
teria,  one  at  a  time.  The  process  proceeds  as  shown  in 
Figure  6-4. 

Having  constructed  the  COST  branch,  the  user  continues 
by  creating  the  MISC  branch.  That  exchange  is  shown  in 
Figure  6-5.  Note  that  the  second-level  criteria  are  attached 
automatically  to  all  of  the  first-level  criteria.  Similarly, 
if  there  were  third- level  criteria  they  would  be  attached  to 
all  of  the  second- level  criteria. 
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ENTER  THE  CRITERIA,  1  PER  LINE,  AFTER  THE  "#"  SIGN. 

THE  DIN  MUST  BE  SEPARATED  FROM  THE  CRITERION  NAME  BY  A  COMMA. 
THE  CRITERION  NAME  IS  LIMITED  TO  10  CHARACTERS  AND  MUST 
CONTAIN  NO  COMMAS. 

♦  :1, INITIAL 
#: 2, ANNUAL" 

t  s  1 . 1 , DOWN  PAY 

♦  :T^  2, TOT.  PRICE 

# si. 2.1, BASIC  ACFT 
#:1.2.2, AVIONICS 
# : 1 . 2 . 3 , INTERIOR 

♦  :2T1  '/OfrgSKTHfff 

#: 2. 2, FIXED 
#: 2. 1.1, FUEL 
# : 2 . 1 . 2,maint. 

#: 2. 1.2.1, AIRFRAME 

♦  :  2.1”.  2. 2, ENGINE 
#: 2. 1.2. 3, AVIONICS 
# : 2. 2.1, CREW-INS . 

# :2. 2. 2 , OWNERSH I P 
#: 


Figure  6-4 

CONSTRUCTING  THE  "COST"  BRANCH 


ENTER  THE  BRANCH  IDENTIFICATION  NUMBER:  2 

IS  THIS  A  NEW  BRANCH?  YES 

IS  THIS  A  SYMMETRIC  BRANClT?  YES 

HOW  MANY  LEVELS  ARE  TO  APPEAR  IN  THIS  ERANCH?  2 

ENTER  THE  LABELS  FOR  LEVEL  1,  LIMITED  TO  10  CHARACTERS  AND 
CONTAINING  NO  COMMAS: 

1)  COMFORT 

2)  CARGO 

3) 

ENTER  THE  LABELS  FOR  LEVEL  2,  LIMITED  TO  10  CHARACTERS  AND 
CONTAINING  NO  COMMAS: 

1)  SPACE 

2)  QUALITY 

3) 

ENTER  THE  BRANCH  IDENTIFICATION  NUMBER: 


Figure  6-5 

CONSTRUCTING  A  SYMMETRIC  BRANCH 


Having  constructed  the  COST  and  MISC  branches,  the  user 
now  constructs  the  remainder  of  the  evaluation  structure 
using  the  "Create  or  Add  to  a  Structure"  function.  Note 
that  the  entire  structure  could  have  been  created  using  this 
function  alone.  The  branches  were  created  separately  only 
to  illustrate  the  use  of  the  branch  function.  Note  also  how 
the  two  branches  are  attached  to  the  structure.  The  process 
is  shown  in  Figure  6-6. 

Assume  that  after  a  lapse  of  time  the  user  decides  that 
the  miscellaneous  criterion  is  insufficiently  detailed  and 
should  be  expanded.  Assume  that  the  user  has  decided  to 
replace  it  with  an  entirely  new  branch  entitled  INTERIOR,  as 
shown  in  Figure  6-7. 

To  modify  the  existing  structure,  the  user  must  first 
remove  (prune)  the  portion  of  the  structure  that  currently 
exists  under  criterion  1.3,  MISC.  The  pruning  process  is 
shown  in  Figure  6-8. 

Note  that  criterion  1.3,  MISC,  itself  has  not  been 
removed  from  the  structure,  only  the  six  lower-level  cri¬ 
teria  attached  to  it  have  been  removed.  Since  the  user 
wants  to  change  the  name  of  criterion  1.3  from  MISC  to 
INTERIOR,  the  user  invokes  the  "Edit  Structure"  function. 
That  process  is  shown  in  Figure  6-9. 

Note  also  that  the  "Edit  Structure"  function  can  be 
used  to  remove  a  criterion  from  the  structure.  The  process 
is  identical  to  that  shown  in  Figure  6-9  except  that  blank 
spaces  are  entered  for  the  node  label. 
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SELECTION:  3 

IS  THIS  A  NEW  MODEL?  YES 

ENTER  THE  LABELS  FOR  THE  SYSTEMS  TO  BE  EVALUATED , 

ONE  PER  LINE,  LIMITED  TO  3  CHARACTERS. 

1)  A 

2)  B 

3)  C 

4)  D 

5)  E 

6)  “ 

ENTER  THE  CRITERIA,  1  PER  LINE,  AFTER  THE  "#"  SIGN.  THE 
DIN  MUST  BE  SEPARATED  FROM  THE  CRITERION  NAME  BY  A  COMMA 
THE  CRITERION  NAME  IS  LIMITED  TO  10  CHARACTERS  AND  MUST 
CONTAIN  NO  COMMAS. 

#:1, AIRCRAFT 
♦: 1.1, COST 
# :  Y.  2 ,  PERFORMANC 
#:1. 3,MlSc 
* :*BRANCH  1  TO  1.1 
#: 1.2.1, GROUND 
#:1.2.2,TKOFF/LAND 
1:1.2. 3, CRU I SE 
♦ :  Y.  2 . 4 , NAV  EQUIP 
#:1.2. 1.1, REFUEL 
#:1.2.1.2,PRFLT 

# : 1. 2. 2. 1, TK OFF/DI S 

♦  :TT2 .2.2, LAND/pT5~ 

# :  1 .  2 . 2 . 3 , RATE  OF  CL 

♦  :  1 .  2 . 3 .  ITALY!  TUfcfe" 

# :1. 2. 3. 2, SPEED 
♦:1.2.3.3,ftANGE+45MN 
#:172.3.1.1,norm  cruis 
»:Y.2.3.1.2,t  In IT  ALT 
♦ : 1. 2 . 3. 1 . 3 , MAX  ALT 

#:1.2.3.2.i,mAx  cRuise 
»:1.2,3.2.2,EtON  CRUfS 
* :1. 2 . 4. 1, PT.  TO  PT. 

#:l^Tr.2,WRM!^AL  “ 

I:  *  Branch  2  to  i.Y 


Figure  6-6 
CREATING  THE  MODEL 


1.3  INTERIOR 


1.  COMFORT 


2.  CARGO 


r 


i 


l 


r 


l 


1.  NOISE 

LEVEL 

2.  SPACE 

,  1 

1.  LUGGAGE 

1 

2.  FREIGHT 

,  | 

1.  WALK 

I 

2.  SEATING  3.  DESK 

4.  GALLEY 

1.  PRES¬ 

2.  TOTAL 

1.  FLEX  2.  SPACE 

AROUND 

TOILET 

SURIZED 

CUFT 

■  CUFT 

1.  FLOOR  2.  NETS  3.  LITTERS 


Figure  6-7 

INTERIOR  CRITERIA 


SELECTION:  7 

*****  prune  SECTION  ***** 

ENTER  DIN  OF  SECTION  TO  BE  PRUNED:  1^3 
ALL  NODES  AFTER  THE  FOLLOWING  WILL  BE  ERASED 

1.3  MISC 

IS  THAT  CORRECT?  YES 
6  NODES  HAVE  BEEN  REMOVED. 

ENTER  DIN  OF  SECTION  TO  BE  PRUNED: 


Figure  6-8 

PRUNING  THE  STRUCTURE 


SELECTION:  2 

*****  EDIT  STRUCTURE  ***** 

ENTER  DIN  OF  NODE  TO  BE  EDITED:  1.3 

1.3  MISC 

EDITED  NODE: 


1.3  INTERIOR 


ENTER  THE  CRITERIA,  1  PER  LINE,  AFTER  THE  "#"  SIGN. 

THE  DIN  MUST  BE  SEPARATED  FROM  THE  CRITERION  NAME  BY  A  COMMA 
THE  CRITERION  NAME  IS  LIMITED  TO  10  CHARACTERS  AND  MUST 
CONTAIN  NO  COMMAS. 

* 

* 

# 

* 

# 

# 

* 

* 

* 

* 

« 

# 

* 

# 

# 

* 

* 

* 


Figure  6-10 

ADDING  TO  THE  STRUCTURE 


Before  saving  the  newly  created  model,  the  user  may 
wish  to  examine  the  computer  version  of  the  structure  to 
ensure  that  all  of  the  desired  nodes  exist  and  that  all  of 
the  undesired  nodes  have  been  removed.  It  is  also  wise  to 
check  the  structure  before  calling  "Edit  Structure"  to 
determine  what  changes  need  to  be  made.  In  both  cases,  the 
user  selects  "Print  Review  Sheet”  and  receives  a  printed 
listing  of  the  model  structure.  When  "Print  Review  Sheet" 
is  selected,  the  computer  instructs  the  user  how  to  activate 
the  printer  in  order  to  obtain  the  hard  copy.  Figure  6-11 
depicts  this  display. 


X , tUraUKi 

2, CARGO 

1.1,  NOISE  LEVEL 
1.2, SPACE 

1. 2.1,  WALK  ARND 
1.2.2, SEATING 

1.2. 3,  DESK 

1.2.4, gm7toilet 

2.1,  LUGGAGE 

2. 2,  FREIGHT 

2. 1.1,  PRESSURIZED 

2. 1.2,  TOT  CU  FT 

2.2.1, flexiblty 

2.2. 2, space/duft 

2. 2. 1.1,  FLOOR 

2. 2. 1.2,  NETS 

2.2.1. 3,  LITTERS 


SELECTION:  8 

PRESS  RED  BUTTON  ON  PRINTER  TO  TURN  IT  ON. 

ALIGN  PAPER  AND  RETURN  CARRIAGE  TO  OBTAIN  HARD  COPY. 

1  AIRCRAFT 

1.1  COST 

1.1.1  INITIAL 

1.1. 1.1  DOWN  PAY 

1.1. 1.2  TOT.  PRICE 

1.1.1. 2.1  BASIC  ACFT 

1.1. 1.2. 2  AVIONICS 

1.1. 1.2. 3  INTERIOR 
1.1.2  ANNUAL 

1.1. 2.1  OPERATING 

1.1. 2. 1.1  FUEL 

1.1. 2. 1.2  MAINT. 

1.1. 2. 1.2.1  AIRFRAME 

1.1. 2. 1.2. 2  ENGINE 

1.1. 2. 1.2. 3  AVIONICS 

1.1. 2. 2  FIXED 

1.1. 2. 2.1  CREW- INS. 

1.1. 2. 2. 2  OWNERSHIP 

1.2  PERFORMANC 

1.2.1  GROUND 

1. 2.1.1  REFUEL 

1.2. 1.2  PRFLT/STRT 

1.2.2  TKOFF/LAND 

1.2. 2.1  TKOFF/DIS 

1.2. 2. 2  LAND/DIS 

1.2. 2. 3  RATE  OF  CL 

1.2.3  CRUISE 

1.2. 3.1  ALTITUDE 

1.2. 3.1.1  NORM  CRUIS 

1.2. 3. 1.2  T  INIT  ALT 

1.2. 3. 1.3  MAX  ALT 

1.2. 3. 2  SPEED 

1.2. 3. 2.1  MAX  CRUISE 

1.2. 3. 2. 2  ECON  CRUISE 

1 . 2 . 3 . 3  RANGE+4  5MN 
1.2.4  NAV  EQUIP 

1. 2.4.1  PT.  TO  PT. 

1.2. 4. 2  TERMINAL 

1.3  INTERIOR 

1.3.1  COMFORT 

1. 3.1.1  NOISE  LEVL 


Figure  6-11 
PRINT  REVIEW  SHEET 
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1.3. 1.2  SPACE 

1.3. 1.2.1  WALK  ARND 

1.3. 1.2. 2  SEATING 

1.3. 1.2. 3  DESK 

1.3. 1.2. 4  GAL/TOILET 
1.3.2  CARGO 

1. 3.2.1  LUGGAGE 

1.3. 2. 1.1  PRESURIZED 

1.3. 2. 1.2  TOT  CU  FT 

1.3.2. 2  FREIGHT 

1.3. 2. 2.1  FLEXIBLTY 

1.3. 2. 2. 1.1  FLOOR 

1.3. 2. 2. 1.2  NETS 

1.3. 2. 2. 1.3  LITTERS 

1 . 3 . 2 . 2 . 2  SPACE/CUFT 

PRESS  RED  BUTTON  TO  TURN  OFF  PRINTER. 
RETURN  CARRIAGE  TO  RETURN  TO  PRIMARY  MENU. 


Figure  6-11  (Continued) 
PRINT  REVIEW  SHEET 


Before  permanently  saving  the  model,  the  user  must 
first  select  "Develop  Structure."  This  function  allows  the 
computer  to  organize  the  structure  into  a  proper  internal 
format  for  saving  the  model;  a  structure  must  be  developed 
if  the  model  is  to  be  saved.  Selecting  "Develop  Structure" 
produces  an  exchange  similar  to  that  shown  in  Figure  6-12. 


SELECTION:  5 

A  ROUGH  ESTIMATE  OF  THE  TIME  REQUIRED  TO  DEVELOP 
THE  STRUCTURE  IS  2  MINUTES  AND  30  SECONDS. 

STRUCTURE  HAS  BEEN  DEVELOPED. 

PLEASE  RETURN  CARRIAGE  TO  CONTINUE. 


Figure  6-12 

DEVELOPING  THE  STRUCTURE 
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Once  the  structure  is  developed,  the  model  can  be 
saved.  Figure  6-13  depicts  a  typical  exchange  when  "Save 
Model"  is  selected. 


SELECTION:  4 

HAVE  YOU  DEVELOPED  THE  STRUCTURE?  YES 
CURRENT  MODELS: 

PERSONNEL 

RADIOS 

ENTER  THE  MODEL  NAME:  AIRCRAFT 


Figure  6-13 
SAVING  A  NEW  MODEL 


When  the  user  has  finished  building  and  saving  a  new 
model,  there  may  be  a  need  to  edit  a  previously  constructed 
model.  "Load  Model"  may  be  used  to  load  an  existing  model 
into  the  computer  workspace.  Figure  6-14  depicts  this  pro¬ 
cess.  The  computer  will  then  load  the  desired  model  and 
return  once  again  to  the  primary  menu. 


SELECTION:  1 

SELECT  THE  NUMBER  OF  THE  OPTION  YOU  DESIRE. 
CURRENT  MODELS: 

1)  PERSONNEL 

2)  RADIOS 

3)  AIRCRAFT 

SELECTION:  1 


Figure  6-14 
LOADING  A  MODEL 


49 


To  stop  processing,  the  user  must  return  the  carriage 
without  typing  anything  when  faced  with  the  primary  menu. 
Figure  6-15  depicts  this  process. 


SELECT  THE  NUMBER  OF  THE  OPTION  YOU  DESIRE: 

1)  LOAD  MODEL 

2)  EDIT  STRUCTURE 

3)  CREATE  OR  ADD  TO  A  STRUCTURE 

4 )  SAVE  MODEL 

5)  DEVELOP  STRUCTURE 

6)  CREATE  BRANCH 

7)  PRUNE  SECTION 

8)  PRINT  REVIEW  SHEET 
SELECTION : 

ANY  CHANGES  MADE  DURING  THIS  SESSION  ARE  NOT  PERMANENT 
UNLESS  YOU  SAVED  THE  MODEL. 

DO  YOU  WISH  TO  STOP?  YES 


Figure  6-15 
STOPPING  THE  PROGRAM 


Having  structured  the  aircraft  evaluation  model,  the 
user  must  now  enter  utility  scores  and  criteria  weights. 
That  is  done  by  using  the  RUN  subsystem. 

6.4  Using  the  RUN  Subsystem  to  Input  Values  and  View  the 
Results  of  the  Example  Problem 

Having  loaded  the  RUN  program,  the  user  will  be  asked 
to  load  a  model.  No  other  RUN  functions  can  be  performed 
until  a  model  is  loaded.  A  model  is  loaded  as  shown  in 
Figure  6-16. 
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THE  RUN  SOFTWARE  PROGRAM  HAS  BEEN  LOADED. 
SELECT  THE  NUMBER  OF  THE  OPTION  YOU  DESIRE. 
CURRENT  MODELS: 

1)  PERSONNEL 

2)  RADIOS 

3)  AIRCRAFT 

SELECTION:  3 


Figure  6-16 

LOADING  A  MODEL  FOR  USE  WITH  EVAL 


The  computer  will  now  display  the  primary  RUN  menu,  as  shown 
in  Figure  6-17. 


SELECT  THE  NUMBER  OF  THE  OPTION  YOU  DESIRE. 


1) 

DISPLAY  RESULTS 

2) 

SENSITIVITY 

3) 

EDIT  VALUES 

4) 

PRINT  RESULTS 

5) 

LOAD  MODEL 

6) 

SAVE  MODEL 

7) 

NEW  VALUES 

8) 

PRINT  DATA  SHEET 

SELECTION : 

Figure  6-17 

EVAL— THE  PRIMARY  MENU 


In  order  to  obtain  a  convenient  format  for  recording 
utility  scores  and  criterion  weights,  the  user  should  select 
the  "Print  Data  Sheet"  option.  The  computer  will  describe 
how  to  obtain  a  printed  data  sheet  and  one  will  be  produced. 
The  exchange  might  be  similar  to  that,  shown  in  Figure  6-18. 
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SELECTION:  8 

PRESS  RED  BUTTON  ON  PRINTER  TO  TURN  IT  ON. 

ALIGN  PAPER  AND  RETURN  CARRIAGE  TO  OBTAIN  HARD  COPY. 

A  B  C  D  E 

1.  -  AIRCRAFT 

1.1.  -  COST  (WT:  ) 

1.1.1.  -  INITIAL  (WT:  ) 

1.1. 1.1.  -  DOWN  PAY  (WT  )  _  _  _  _ 

1.1. 1.2.  -  TOT.  PRICE  (WT:  ) 

1.1. 1.2.1.  -  BASIC  ACFT  lWT: _ )  _  _  _  _  __ 

1.1. 1.2. 2.  -  AVIONICS  (WT:  )  _  _  _ _ _ 

1.1. 1.2. 3.  -  INTERIOR  (WT: _ )  _  _  _  _  _ 

1.1.2.  -  ANNUAL  (WT:  ) 

1.1. 2.1.  -  OPERATING  “fWT:  ) 

1.1. 2. 1.1.  -  FUEL  (WT: _ 1  _  _  _  _  _ 

1.1. 2. 1.2.  -  MAINT.  (WT:  ) 

1.1. 2. 1.2.1.  -  AIRFRAME  (WT: _ )  _  _  _  _  _ 

1.1. 2. 1.2. 2.  -  ENGINE  (WT:  ) 

1.1. 2. 1.2. 3.  -  AVIONICS  (WT: _ )  _  _  _  _  _ 

1.1. 2. 2.  -  FIXED  (WT:  ) 

1.1. 2. 2.1.  -  CREW- INS  lWT: _ )  _  _  _  _ 

1.1. 2. 2. 2.  -  OWNERSHIP  (WT: _ )  _  _ _ _ 

1.2.  -  PERFORMANC  (WT:  ) 

1.2.1.  -  GROUND  (WT:  HP 

1.2. 1.1.  -  REFUEL  (WT:  )  _  _  _  _  _ 

1.2. 1.2.  -  PRFLT/STRT  (WT:  )  _  _  _  _  _ 

1.2.2.  -  TKOFF/LAND  (WT:  T  ~  ~  ” 

1. 2.2.1.  -  TKOFF/DIS  (WTl  )  _  _  _  _ 

1.2. 2. 2.  -  LAND/DIS  (WT:  )  “  _ 

1.2. 2. 3.  -  RATE  OF  CL  (WT: _ )  _  _  _  _  _ 

1.2.3.  -  CRUISE  (WT:  ) 

1.2. 3.1  -  ALTITUDE  (WFT  ) 

1.2. 3. 1.1.  -  NORM  CRUIS  “lWT:  ) 

1.2. 3.1. 2.  -  T  INIT  ALT  (WT:  ) 

1.2. 3.1. 3.  -  MAX  ALT  (WT:  ) 

1.2. 3. 2.  -  SPEED  (WT: _ ) 

1.2. 3. 2.1.  -  MAX  CRUISE  (WT:  )  _  _ 

1.2. 3. 2. 2.  -  ECON  CRUIS  (WT:  )  _  _ 

1.2. 3. 3.  -  RANGE+45MN  (WT:  ~T~ 

1.2.4.  -  NAV  EQUIP  (WT: _ ) 

1.2. 4.1.  -  PT.  TO  PT.  (WT: _ )  _  _  _  _ 

1.2. 4. 2.  -  TERMINAL  (WT:  ) 

1.3.  -  INTERIOR  (WT: _ ) 

1.3.1.  -  COMFORT  (WT:  ) 

1.3. 1.1.  -  NOISE  LEVL  “lWT:  )  _  _ 

1.3. 1.2.  -  SPACE  (WT:  ) 

1.3. 1.2.1.  -  WALK  ARND  (WT:  )  _  _ 

1.3. 1.2. 2.  -  SEATING  (WT:  ) 


Figure  6-18 

OBTAINING  A  WORK  SHEET 
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.  -  GAL/TOILET  (WT: 
CARGO  (WT:  ) 

-  LUGGAGE  lWT:  ) 

.  -  PRESURIZED  (WT: 
.  -  TOT  CU  FT  (WT: 

-  FREIGHT  (WT:  ) 

.  -  FLEXIBLTY  JWfi 
.1.  -  FLOOR  (WT: 


1.3. 2. 2. 1.3.  -  LITTERS  (WT: _ )  _ 

1.3. 2. 2. 2.  -  SPACE/CUFT  (WT: _ )  _ 

PRESS  RED  BUTTON  TO  TURN  OFF  PRINTER. 
RETURN  CARRIAGE  TO  RETURN  TO  PRIMARY  MENU. 


Figure  6-18  (Continued) 
OBTAINING  A  WORKSHEET 


The  "New  Values"  function  will  allow  those  assessed 
values  to  be  input  to  the  model.  Selecting  "New  Values" 
yields  a  secondary  menu,  as  shown  in  Figure  6-19. 


SELECTION:  7 

SELECT  THE  NUMBER  OF  THE  OPTION  YOU  DESIRE. 

1)  LOAD  SCORES 

2)  LOAD  WEIGHTS 
SELECTION:  1 


Figure  6-19 

SECONDARY  MENU  UNDER  "NEW  VALUES" 


When  "Load  Scores"  is  selected,  the  computer  prints  out  the 
model  criteria  line  by  line,  in  order  by  DIN,  stopping  for 
user  input  of  scores  whenever  a  bottom-level  criterion  is 
reached.  A  utility  score  must  be  input  for  each  of  the  five 
aircraft  being  evaluated.  Figure  6-20  displays  how  this 


interchange  might  appear.  When  the  user  has  finished  in¬ 
putting  the  scores,  the  computer  will  return  to  the  sec¬ 
ondary  menu. 


SELECTION :  1 

,  A  B  C  D  E 

1.  -  AIRCRAFT 


1.1.  -  COST 

1.1.1.  -  INITIAL 

1.1. 1.1.  -  DOWN  PAY 

75 

0 

50 

85 

100 

1.1. 1.2.  -  TOT.  PRICE 

1 1 

1.1. 1.2.1.  -  BASIC  ACFT 

70 

0 

50 

80 

100 

1.1. 1.2. 2.  -  AVIONICS 

55 

75 

155 

55 

~ 

1.1. 1.2. 3.  -  INTERIOR 

55 

155 

"75 

55 

m 

1.1.2.  -  ANNUAL 

' 

1.1. 2.1.  -  OPERATING 

1.1. 2. 1.1.  -  FUEL 

75 

40 

0 

80 

100 

1.1. 2. 1.2.  -  MAINT. 

^ 1X1 

1.1. 2. 1.2.1.  -  AIRFRAME 

60 

0 

45 

100 

80 

1.1. 2. 1.2. 2.  -  ENGINE 

55 

55 

0 

“55 

155 

1.1. 2. 1.2. 3.  -  AVIONICS 

75 

"5 

35 

155 

“55 

1.1. 2. 2.  -  FIXED 

. . 

1.1. 2. 2.1.  -  CREW- INS 

60 

0 

60 

100 

80 

1.1. 2. 2. 2.  -  OWNERSHIP 

75 

"5 

35 

155 

n 

1.2.  -  PERFORMANC 

" 

1.2.1.  -  GROUND 

1.2. 1.1.  -  REFUEL 

100 

0 

60 

80 

80 

1.2. 1.2.  -  PRFLT/STRT 

“55 

155 

Ji 

35 

55 

1.2.2.  -  TKOFF/LAND 

" 

1.2. 2.1.  -  TKOFF/DIS 

60 

0 

55 

70 

100 

1.2. 2. 2.  -  LAND/DI S 

75 

75 

"5 

55 

155 

1.2.2. 3.  -  RATE  OF  CL 

55 

155 

n 

J[ 

“55 

1.2.3.  -  CRUISE 

~~ 

1.2. 3.1.  -  ALTITUDE 

1.2. 3. 1.1.  -  NORM  CRUIS 

75 

95 

100 

0 

30 

1.2. 3. 1.2.  -  T  INIT  ALT 

55 

55 

155 

“5 

15 

1.2. 3. 1.3.  -  MAX  ALT 

75 

155 

“55 

35 

1.2. 3. 2.  -  SPEED 

**  ™ 

1.2. 3. 2.1.  -  MAX  CRUISE 

70 

100 

85 

0 

35 

1.2. 3. 2. 2.  -  ECON  CRUIS 

75 

155 

55 

“5 

15 

1.2. 3. 3.  -  RANGE+45MN 

55 

155 

75 

“5 

75 

Figure  6-20 
LOADING  SCORES 


1.2.4.  -  NAV  EQUIP 


1.2. 4.1.  -  PT.  TO  PT. 

60 

100 

80 

40 

0 

1.2. 4. 2.  -  TERMINAL 

H 

m 

H 

m 

1.3.  -  INTERIOR 

1.3.1.  -  COMFORT 

1.3. 1.1.  -  NOISE  LEVL 

_50 

100 

70 

_0 

45 

1.3. 1.2.  -  SPACE 

1.3. 1.2.1.  -  WALK  ARND 

40 

70 

0 

100 

70 

1.3. 1.2. 2.  -  SEATING 

7T 

lffff 

“ff 

“5T 

T? 

1.3. 1.2. 3.  -  DESK 

7j> 

TM 

5(5 

80 

1.3. 1.2. 4.  -  GAL/TOILET 

TM 

ST 

15 

15 

1.3.2.  -  CARGO 

1.3. 2.1.  -  LUGGAGE 

1.3. 2. 1.1.  -  PRESURIZED 

50 

100 

40 

0 

85 

1.3. 2. 1.2.  -  TOT  CU  FT 

15 

T5o 

_0 

70 

80 

1. 3.2.2.  -  FREIGHT 

1.3. 2. 2.1.  -  FLEXIBLTY 

1.3. 2. 2. 1.1.  -  FLOOR 

0 

40 

50 

80 

100 

1.3. 2. 2. 1.2.  -  NETS 

lffo 

TO 

H5 

IT 

"To 

1.3. 2. 2. 1.3.  -  LITTERS 

T 

Tff 

7T 

1M 

1.3. 2. 2. 2.  -  SPACE/CUFT 

_0 

m 

6T 

T5- 

“65 

Figure  6-20  (Continued) 
LOADING  SCORES 


The  "Load  Weights"  function  allows  the  user  to  input 
the  relative  weights  of  the  criteria  branching  from  each 
node.  The  computer  presents  the  criteria  to  the  user,  one 
by  one,  in  order  by  DIN,  requesting  the  relative  weights  of 
the  successive  criteria  and  requesting  confirmation  for  the 
normalized  values.  The  process  might  proceed  as  shown  in 
Figure  6-21. 


After  the  final  weights  have  been  input  and  confirmed, 
the  computer  returns  the  user  to  the  secondary  menu.  The 
user  may  then  return  the  carriage  without  selecting  either 
"Load  Weights"  or  "Load  Scores"  to  return  to  the  primary 
menu. 
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SELECTION:  2 

1  AIRCRAFT 

CURRENT: 
NEW  WTS : 
NORMALIZED 


COST  PERFORMANCE  INTERIOR 

0  0  0 

6  10  4 

30  50  20 


IF  THESE  ARE  CORRECT  TYPE  GO: 

NEW  WTS :  6 

NORMALIZED:  35 

IF  THESE  ARE  CORRECT  TYPE  GO:  GO 


1.1  COST 

CURRENT: 
NEW  WTS: 
NORMALIZED: 


INITIAL 

0 

35 

35 


ANNUAL 

0 

65 

65 


IF  THESE  ARE  CORRECT  TYPE  GO:  GO 


1.1.1  INITIAL 

CURRENT : 
NEW  WTS: 
NORMALIZED: 


DOWN  PAY 
0 
2 


TOT.  PRICE 
0 
3 

So 


IF  THESE  ARE  CORRECT  TYPE  GO:  GO 


1.1. 1.2  TOT.  PRICE 

I 

CURRENT: 

NEW  WTS: 
NORMALIZED: 


BASIC  ACFT 
0 

60 

sir 


AVIONICS 

0 

15 

13 


INTERIOR 

0 

25 

53 


IF  THESE  ARE  CORRECT  TYPE  GO:  GO 


1.1.2  ANNUAL 

CURRENT: 
NEW  WTS: 
NORMALIZED: 


OPERATING 

0 

1 

35 


FIXED 

0 

1 

35 


IF  THESE  ARE  CORRECT  TYPE  GO:  GO 


Figure  6-21 
INPUTTING  WEIGHTS 
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1.1. 2.1  OPERATING 


CURRENT: 

FUEL 

0 

MAINT. 

0 

NEW  WTS: 

1 

1 

NORMALIZED: 

01 

01 

IF  THESE  ARE  CORRECT 

TYPE  GO: 

GO 

1.1. 2. 1.2  MAINT. 

AIRFRAME 

CURRENT :  0 

ENGINE 

0 

AVIONICS 

o 

NEW  WTS: 
NORMALIZED: 

25 

TO 

45 

TO 

30 

30 

IF  THESE  ARE  CORRECT 

TYPE  GO: 

GO 

1.1. 2. 2  FIXED 

CREW- INS 

CURRENT:  0 

OWNERSHIP 

0 

NEW  WTS: 

3 

7 

NORMALIZED: 

31 

75 

IF  THESE  ARE  CORRECT  TYPE  GO:  GO 


1.2  PERFORMANC 

CURRENT: 
NEW  WTS: 
NORMALIZED: 


GROUND 


TKOFF/LAND 

0 

40 

TO 


CRUISE 

0 

35 

31 


NAV  EQUIP 
0 
15 
15 


IF  THESE  ARE  CORRECT  TYPE  GO: 
1.2.1  GROUND 


CURRENT : 
NEW  WTS: 
NORMALIZED: 


REFUEL 

0 

3 

70 


IF  THESE  ARE  CORRECT  TYPE  GO: 
1.2.2  TKOFF/LAND 


CURRENT: 
NEW  WTS: 
NORMALIZED: 


TKOFF/DIS 

0 

3 

31 


GO 


PRFLT/STRT 

0 

7 

70- 


go 


LAND/DI S 
0 
5 

01 


IF  THESE  ARE  CORRECT  TYPE  GO:  GO 


RATE  OF  CL 


Figure  6-21  (Continued) 
INPUTTING  WEIGHTS 


1.2.3  CRUISE 

CURRENT: 
NEW  WTS: 
NORMALIZED: 


ALTITUDE 

0 

2 

75 


IF  THESE  ARE  CORRECT  TYPE  GO: 


1.2. 3.1  ALTITUDE 

CURRENT : 

NEW  WTS: 
NORMALIZED: 


NORM  CRUIS 
0 
6 
77 


IF  THESE  ARE  CORRECT  TYPE  GO: 


1.2. 3.2  SPEED 

CURRENT: 
NEW  WTS: 
NORMALIZED: 


MAX  CRUISE 
0 
35 
35 


IF  THESE  ARE  CORRECT  TYPE  GO: 


1.2.4  NAV  EQUIP 

CURRENT: 

NEW  WTS: 
NORMALIZED: 


PT.  TO  PT. 
0 
2 
T5 


IF  THESE  ARE  CORRECT  TYPE  GO: 


1.3  INTERIOR 

CURRENT: 
NEW  WTS: 
NORMALIZED: 


COMFORT 

0 

4 

17 


IF  THESE  ARE  CORRECT  TYPE  GO: 


1.3.1  COMFORT 

CURRENT: 
NEW  WTS: 
NORMALIZED: 


NOISE  LEVL 
0 
3 

77 


IF  THESE  ARE  CORRECT  TYPE  GO: 


SPEED 

0 

3 

37 


T  INIT  ALT 
0 
3 
37 


ECON  CRUISE 
0 
65 
65 


TERMINAL 

0 

3 

77 


CARGO 

0 

1 

77 


SPACE 

0 

2 

T7 


RANGE+45MN 

0 

5 

77 


Figure  6-21  (Continued) 
INPUTTING  WEIGHTS 


MAX  ALT 
0 
1 
17 
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1.3. 1.2  SPACE 


WALK  ARND 

SEATING 

DESK 

CURRENT: 

0 

0 

0 

NEW  WTS : 

5 

10 

3 

NORMALIZED: 

IT 

TT 

IT 

IF  THESE  ARE 

CORRECT  TYPE  GO: 

GO 

1.3.2  CARGO 

CURRENT: 

LUGGAGE 

0 

FREIGHT 

0 

NEW  WTS: 

7 

3 

NORMALIZED: 

7T 

Iff 

IF  THESE  ARE 

CORRECT  TYPE  GO: 

GO 

1. 3.2.2  FREIGHT 

FLEXIBLTY 

SPACE/CUFT 

CURRENT: 

0 

0 

NEW  WTS: 

45 

55 

NORMALIZED: 

TS 

S3 

IF  THESE  ARE 

CORRECT  TYPE  GO: 

GO 

1.3. 2.2.1  FLEXIBLTY 

FLOOR 

NETS  LITTERS 

CURRENT: 

0 

0 

0 

NEW  WTS: 

60 

25 

15 

NORMALIZED: 

60 

IT 

IT 

IF  THESE  ARE 

CORRECT  TYPE  GO: 

GO 

Figure  6-21 

INPUTTING 

(Continued) 

WEIGHTS 

While  reviewing  the  numbers  that  had  been  input,  the 
user  realized  that  several  errors  had  been  made  while  input¬ 
ting  the  values.  Therefore,  "Edit  Values"  was  selected, 
presenting  the  secondary  menu  depicted  in  Figure  6-22. 
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SELECTION :  3 

SELECT  THE  NUMBER  OF  THE  OPTION  YOU  DESIRE. 

1)  EDIT  SCORES 

2)  EDIT  WEIGHTS 

SELECTION:  1 


Figure  6-22 

SECONDARY  MENU  BENEATH  "EDIT  VALUES" 


Next,  "Edit  Scores"  was  selected  and  the  process  shown  in 
Figure  6-23  occurred. 


SELECTION:  1 

*****  EDIT  SCORES  ***** 

DIN  TO  BE  EDITED:  1.2. 1.1 

1.2. 1.1  REFUEL 

ABC 
CURRENT  SCORES:  100  0  60 
NEW  SCORES:  100  60  _0 

DIN  TO  BE  EDITED: 


Figure  6-23 
EDITING  SCORES 


Next,  the  "Edit  Weights"  function  was  selected  and  the 
exchange  shown  in  Figure  6-24  ensued.  The  user  eventually 
returns  the  carriage  to  signify  that  the  editing  of  weights 
has  been  completed.  This  will  place  the  user  at  the  sec¬ 
ondary  menu.  Returning  the  carriage  without  selecting 
either  "Edit  Scores"  or  "Edit  Weights"  returns  the  user  to 
the  primary  menu. 


D  E 

80  80 

80  80 


SELECTIONS  2 

*****  eBit  WEIGHTS  ***** 
DIN  TO  BE  EDITED:  1.1.2 


1.1.2  ANNUAL 

CURRENT : 
NEW  WTS: 
NORMALIZED: 


OPERATING 

50 

_3 

60 


FIXED 

50 

2 

40 


IF  THESE  ARE  CORRECT  TYPE  GO:  GO 
DIN  TO  BE  EDITED: 


Figure  6-24 
EDITING  WEIGHTS 


In  order  to  avoid  losing  the  model  inadvertently,  now 
that  values  have  been  added  to  it,  the  user  should  immediately 
save  the  model.  When  "Save  Model"  is  selected,  an  exchange 
similar  to  that  shown  in  Figure  6-25  occurs.  Once  the  model 
is  saved,  the  computer  will  automatically  return  the  user  to 
the  primary  menu. 


SELECTION:  6 

*****  save  MODEL  ***** 

CURRENT  MODELS 
PERSONNEL 
RADIOS 
AIRCRAFT 

ENTER  NAME  OF  NEW  MODEL:  AIRCRAFT 
A  MODEL  WITH  THIS  NAME  ALREADY  EXISTS. 
DO  YOU  WISH  TO  REPLACE  IT?  YES 


Figure  6-25 

SAVING  A  MODEL  AFTER  ENTERING  VALUES 


In  order  to  view  the  results  of  the  model,  the  user 
should  select  "Display  Results."  An  exchange  similar  to 
that  shown  in  Figure  6-26  might  result. 


SELECTION :  1 

*****  DISPLAY  RESULTS  ***** 
DIN  TO  BE  DISPLAYED:  1 

1  AIRCRAFT 


FACTOR 

WT 

A 

B 

C 

D 

E 

CUMWT 

1) 

COST 

(30) 

68 

35 

47 

81 

82 

30.00% 

2) 

PERFORMANC 

(55) 

68 

81 

69 

47 

64 

55.00% 

3) 

INTERIOR 

(15) 

52 

91 

55 

43 

60 

15.00% 

TOTAL 

66 

68 

61 

57 

69 

100.00% 

TO 

BE  DISPLAYED: 

1.2 

AIRCRAFT  -  PERFORMANCE 

FACTOR 

WT 

A 

B 

C 

D 

E 

CUMWT 

1) 

GROUND 

(10) 

85 

81 

27 

52 

59 

5.50% 

2) 

TKOFF/LAND 

(40) 

65 

66 

66 

66 

88 

22.00% 

3) 

CRUISE 

(35) 

65 

91 

80 

32 

51 

19.25% 

4) 

NAV  EQUIP 

(15) 

69 

94 

83 

31 

35 

8.25% 

TOTAL 

68 

81 

69 

[  7 

64 

55.00% 

DIN  TO  BE  DISPLAYED: 


Figure  6-26 
DISPLAY  RESULTS 


The  user  could  examine  each  and  every  DIN  if  desired.  Here, 
though,  the  user  decided  to  examine  more  closely  DIN  "1.2 
PERFORMANCE."  The  "Sensitivity"  function  was  chosen  in 
order  to  closely  examine  this  factor. 

Selecting  "Sensitivity"  causes  the  computer  to  request 
the  DIN  of  the  factor  to  be  examined.  The  computer  will 
then  request  the  minimum  and  maximum  cumulative  weights  to 
be  considered.  Scores  for  each  alternative  will  be  li-sted, 
assuming  that  the  specified  factor  has  anywhere  between  the 
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minimum  and  maximum  cumulative  weight  (by  increments  of  10% 
of  the  difference  between  the  highest  and  lowest  cumulative 
weights  considered).  Figure  6-27  depicts  the  results  of 
selecting  "Sensitivity." 


SELECTION:  2 

*****  SENSITIVITY  ***** 

WHAT  FACTOR  DO  YOU  WISH  TO  EXAMINE? 
ENTER  DIN:  1.2 

1.2  PERFORMANC 

CURRENT  CUMWT :  55.00 

MINIMUM  CUMWT:  0 
MAXIMUM  CUMWT:  100 

1.2  PERFORMANCE  CURRENT  CUMWT :  55.00 


WEIGHT 

A 

B 

C 

D 

E 

.0 

63 

54 

50 

68 

74* 

10.0 

63 

56 

52 

66 

73* 

20.0 

64 

59 

54 

64 

72* 

30.0 

64 

62 

56 

62 

71* 

40.0 

65 

64 

58 

60 

70* 

50.0 

65 

67 

60 

58 

69* 

60.0 

66 

70* 

62 

56 

68 

70.0 

66 

72* 

63 

54 

67 

80.0 

67 

75* 

65 

52 

66 

90.0 

67 

78* 

67 

49 

65 

100.0 

68 

81* 

69 

47 

64 

WHAT  FACTOR  DO  YOU  WISH  TO  EXAMINE? 
ENTER  DIN: 


Figure  6-27 
SENSITIVITY 


Note  that  aircraft  E  is  optimal  until  "1.2  PERFORMANCE " 
receives  about  50%  of  the  cumulative  weight?  "1.2  PERFOR¬ 
MANCE"  could  be  examined  even  more  closely  (and  a  threshold 
value  obtained)  by  selecting  and  specifying  minimum  and 
maximum  cumulative  weights  of  50%  and  60%.  In  addition, 
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other  sensitivity  analyses  could  be  performed  for  any  of  the 
individual  criteria  in  the  diagram. 

Once  the  model  has  been  displayed  and  sensitivity 
analyses  performed,  the  user  may  wish  to  alter  the  model. 
"Edit  Values"  should  be  used  if  any  permanent  changes  are 
desired.  After  the  changes  are  made,  the  user  must  once 
again  select  "Save  Model"  to  record  these  changes  perma¬ 
nently. 

Selecting  "Print  Results"  will  produce  a  printout  of 
all  matrices  available  through  "Display  Results,"  in  order 
by  DIN.  Figure  6-28  demonstrates  its  use  for  the  first 
three  nodes  only. 


SELECTION:  £ 

PRESS  RED  BUTTON  ON  PRINTER  TO  TURN  IT  ON. 

ALIGN  PAPER  AND  RETURN  CARRIAGE  TO  OBTAIN  HARD  COPY. 


1.  OVERALL 


FACTOR 

WT 

A 

B 

C 

D 

E 

CUMWT 

1) 

COST 

(30) 

68 

32 

47 

81 

82 

30.00% 

2) 

PERFORMANC 

(55) 

68 

81 

69 

47 

64 

55.00% 

3) 

INTERIOR 

(15) 

52 

91 

55 

43 

60 

15.00% 

TOTAL 

66 

68 

61 

57 

69 

100.00% 

1.1. 

COST 

FACTOR 

WT 

A 

B 

C 

D 

E 

CUMWT 

1) 

INITIAL 

(35) 

70 

25 

56 

70 

75 

10.50% 

2) 

ANNUAL 

(65) 

67 

36 

43 

87 

86 

19.50% 

TOTAL 

68 

32 

47 

81 

82 

30.00% 

1.1. 

1.  INITIAL 

FACTOR 

WT 

A 

B 

C 

D 

E 

CUMWT 

1) 

DOWN  PAY 

*(40) 

75 

0 

50 

85 

100 

4.20% 

2) 

TOT.  PRICE 

(60) 

66 

41 

60 

60 

59 

6.30% 

TOTAL 

70 

25 

56 

70 

75 

10.50% 

Figure  6-28 
PRINT  RESULTS 


Having  finished  with  the  "AIRCRAFT”  model,  the  user  may  wish 
to  exercise  another  model.  "Load  Model"  could  be  used  to 
gain  access  to  it.  On  the  other  hand,  the  user  may  wish  to 
end  the  session.  If  so,  the  carriage  should  be  returned 
without  selecting  an  option  from  the  primary  menu.  The 
computer  will  then  remind  the  user  that  any  changes  made 
during  the  session  will  be  saved  only  if  "Save  Model"  has 
been  selected.  The  computer  then  requests  confirmation  that 
the  user  is  finished  before  actually  terminating  the  program 


7.0  ABRIDGED  USERS  MANUAL 


This  section  is  designed  for  the  user  who  is  already 
familiar  with  EVAL.  It  describes  the  essential  elements  of 
the  evaluation  procedure ,  and  discusses  how  those  are  molded 
into  the  multi-attribute  utility  model  implemented  by  EVAL. 

7.1  Structuring  the  Decision  Problem 

Every  decision  problem  appropriate  for  EVAL  includes 
the  following  elements: 

o  A  list  of  alternative  systems  to  be  evaluated; 

o  An  overall  evaluation  criterion;  and 

o  A  hierarchical  set  of  criteria  into  which  the 
overall  criterion  is  decomposed. 

Judgmental  assessments  which  must  be  made  include: 

o  The  relative  importance  weights  of  the  criteria; 
and 

o  The  relative  utility  of  each  system  evaluated  at 
the  bottom- level  criteria. 

Once  the  elements  and  assessments  are  specified ,  the  user  is 
ready  to  use  the  EVAL  program. 
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7.2  Subsystems 


EVAL  consists  of  two  subsystems:  STRUCTURE  and  RUN. 
STRUCTURE  is  used  to  construct  the  model.  RUN  is  used  to 
enter  values  and  weights  and  to  display  results. 

7.2.1  Functions  available  in  STRUCTURE  -  Once  STRUC¬ 
TURE  is  loaded  into  the  computer,  a  menu  of  functions  appears 
as  follows: 

o  Load  Model 
o  Edit  Structure 
o  Create  or  add  to  a  Structure 
o  Save  Model 
o  Develop  Structure 
o  Create  Branch 
o  Prune  Section 
o  Print  Review  Sheet. 

The  user  must  first  either  load  an  existing  model  or  create 
a  new  one. 


Selecting  "Load  Model"  allows  the  user  to  load 
an  existing  model.  Once  a  model  is  loaded,  the  user  may 
select  "Edit  Structure"  in  order  to  edit  the  existing  struc¬ 
ture  by  changing  the  criteria  names  or  data  identification 
numbers  (DINs) ,  or  by  deleting  single  criteria.  Selecting 
"Create  or  Add  to  a  Structure"  allows  the  user  to  change  the 
model  by  adding  new  criteria.  Whole  sections  of  criteria 
may  be  deleted  by  using  "Prune  Section."  Whole  sections  of 
criteria  may  be  created  using  "Create  Branch"  (these  must  be 
added  to  the  structure  later  using  "Create  or  Add  to  a 
Structure").  After  any  changes  are  made,  "Develop  Struc¬ 
ture"  must  be  selected  to  properly  format  the  model  inter¬ 
nally  for  saving.  The  model  can  then  be  saved  using  the 


"Save  Model"  function.  A  printout  can  be  obtained  using 
"Print  Review  Sheet." 

To  create  an  entirely  new  model,  the  user  selects 
"Create  or  Add  to  a  Structure."  The  names  of  the  alternative 
systems  to  be  evaluated  are  typed  in,  and  the  DINs  and  cri¬ 
teria  names  are  input  one  by  one.  Branches  may  be  added  if 
they  have  been  previously  created.  Any  necessary  editing 
and  pruninq  is  done  before  the  model  is  developed  and  saved. 

A  printout  may  be  obtained  using  "Print  Review  Sheet." 

7.2.2  Options  available  in  RUN  -  When  RUN  is  loaded 
into  the  computer,  the  program  will  automatically  ask  the 
user  to  load  a  model  from  the  model  library.  A  menu  of 
options  will  appear,  as  follows: 

o  Display  Results 
o  Sensitivity 
o  Edit  Values 
o  Print  Results 
o  Load  Model 
o  Save  Model 
o  New  Values 
o  Print  Data  Sheet. 

Since  a  model  has  already  been  loaded  from  the 
model  library,  the  user  need  not  select  "Load  Model"  to 
perform  this  function.  "Load  Model"  may  be  used  later  to 
work  on  another  problem. 

The  user  should  select  "Print  Data  Sheet"  to 
obtain  a  convenient  form  on  which  to  assess  and  record 
scores  and  relative  weights  and  to  provide  a  record  of  the 
existing  structure.  If  weights  and  scores  have  not  been 
entered  into  the  model,  "New  Values"  must  be  selected  in 
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order  to  accomplish  that  function.  If  any  typographical 
errors  were  made  in  entering  the  data,  "Edit  Values"  may  be 
selected  to  correct  them.  If  any  additions  or  changes  have 
been  made  to  the  data,  the  model  should  now  be  saved  by 
using  "Save  Model." 

Once  the  values  have  been  correctly  loaded  into 
the  model,  the  user  may  wish  to  examine  the  results.  Select 
ing  "Display  Results"  causes  the  computer  to  display  the 
normalized  weights,  cumulative  weights,  and  intermediate 
scores  for  the  DIN  designated  by  the  user,  and  for  the 
criteria  directly  attached  to  that  DIN.  The  user  may  choose 
to  examine  any  number  of  DINs.  Once  the  user  is  satisfied, 
"Sensitivity"  may  be  chosen  to  examine  the  threshold  cumula¬ 
tive  weight  for  any  specific  DIN.  A  threshold  weight  is  the 
cumulative  weight  which  causes  a  new  system  to  replace  the 
previously  highest  scoring  system.  Any  number  of  DINs  may 
be  examined  using  "Sensitivity." 

After  examination  of  the  model,  the  user  may 
wish  to  permanently  alter  some  of  the  weights  and  scores. 

If  so,  the  "Edit  Values"  function  should  be  requested  and 
the  changes  made.  The  model  should  be  saved,  and  a  printout 
of  the  final  results  obtained  by  using  the  "Print  Results" 
function  which  prints  out,  in  order  of  DIN,  all  of  the  cri¬ 
terion  matrices  available  in  "Display  Results.”  Another 
model  may  then  be  loaded  for  examination  by  using  "Load 
Model"  or  the  computer  session  may  be  terminated. 


